A fault node elimination method and device, electronic equipment and storage medium
By receiving faulty replica anomaly information reported by the client or the primary replica, sending probe commands to normal replicas and determining which to remove, the problem of failing to remove faulty replicas in a timely manner in the existing technology is solved, realizing automated faulty replica processing and improving data processing efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA CONSTRUCTION BANK
- Filing Date
- 2023-08-02
- Publication Date
- 2026-07-03
Smart Images
Figure CN116781487B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of cloud disk operation and maintenance management technology, and in particular to a method and device for removing faulty nodes, electronic equipment, and storage media. Background Technology
[0002] To improve the high availability of cloud disks, a three-replica storage mechanism is commonly used for data storage. This involves forming a replica group with three storage nodes, consisting of one primary replica and multiple secondary replicas. To ensure data consistency across all replicas in the group, a write-to-disk success message is only returned after all replicas have successfully written the data. Therefore, if a replica in the group experiences a prolonged or permanent failure, it needs to be promptly removed from the cluster, and its data migrated to new replicas.
[0003] Currently, to promptly identify faulty replicas, each replica in the replica group obtains the status of the replicas it is communicating with, and reports the obtained status information of other replicas to the management node every second. To ensure accuracy, if two out of the three replicas in the replica group report a fault in the remaining replica, the management node will monitor that replica within a specified time window. If it continuously receives reports of a fault from that replica, it will remove it from the list.
[0004] However, with the existing method, if a replica fails to communicate with the faulty replica during data writing, it will not report the fault to the management node. Therefore, even if the management node receives an anomaly report from another replica, it will not remove the faulty replica. This results in the failure of both current and subsequent requests. Manual intervention is then required, which is not only costly but also relatively inefficient. Therefore, the existing method cannot effectively guarantee the timely removal of faulty replicas to avoid impacting data processing requests. Summary of the Invention
[0005] In view of the shortcomings of the prior art, this application provides a method and apparatus for removing faulty nodes, an electronic device, and a storage medium to solve the problem that the existing removal methods cannot guarantee the timely removal of faulty copies and avoid affecting the response to data processing requests.
[0006] To achieve the above objectives, this application provides the following technical solution:
[0007] The first aspect of this application provides a method for removing faulty nodes, including:
[0008] Receive exception information about the currently faulty replica reported by the client or the primary replica;
[0009] Send a probe command for the currently faulty replica to the normal replica corresponding to the currently faulty replica, so as to trigger the normal replica corresponding to the currently faulty replica to probe the currently faulty replica; wherein, the normal replica corresponding to the currently faulty replica includes multiple replicas in the replica group to which the currently faulty replica belongs, excluding the currently faulty replica;
[0010] Receive the detection results fed back by the normal replica corresponding to the currently faulty replica;
[0011] Determine whether the detection results fed back by the normal replica corresponding to the current faulty replica all indicate that the current faulty replica is abnormal;
[0012] If it is determined that the detection results fed back by the normal replica corresponding to the current faulty replica all indicate that the current faulty replica is abnormal, then a removal instruction is sent to the current faulty replica to remove the current faulty replica.
[0013] Optionally, in the above-described method for removing faulty nodes, the currently faulty replica is the primary replica, and the abnormal information of the currently faulty replica reported by the client or the primary replica includes:
[0014] The system receives exception information about the primary replica reported by the client; wherein the exception information about the primary replica is generated and fed back to the client when the primary replica fails to write data during the process of responding to a data processing request sent to it by the client.
[0015] Optionally, in the above-described method for removing faulty nodes, the currently faulty replica can be any slave replica, and the abnormal information of the currently faulty replica reported by the client or the primary replica includes:
[0016] The system receives exception information from the slave replicas reported by the primary replica; wherein the exception information from the slave replicas is generated by the slave replicas when they are unable to successfully write data synchronized from the primary replicas to the slave replicas and is fed back to the primary replicas; after responding to a data processing request sent to it by the client and successfully writing the data, the primary replicas synchronize the data to each slave replica in its replica group.
[0017] Optionally, in the above-described fault node removal method, before sending the probe instruction for the currently faulty replica to the normal replica corresponding to the currently faulty replica, the method further includes:
[0018] Based on the anomaly information of the current faulty replica, the routing information of the normal replica corresponding to the current faulty replica is retrieved from the routing information table;
[0019] The step of sending a probe command for the currently faulty replica to the normal replica corresponding to the currently faulty replica includes:
[0020] Based on the routing information of each normal replica corresponding to the current faulty replica, a probe command for the current faulty replica is sent to each normal replica corresponding to the current faulty replica.
[0021] Optionally, in the above-described method for removing faulty nodes, after sending the removal instruction to the current faulty replica, the method further includes:
[0022] Real-time monitoring to see if successful removal information is received from the currently faulty replica;
[0023] When it is detected that the successful removal information of the current faulty copy has been received, the latest copy information is sent to the client to trigger the client to update the recorded copy information using the latest copy information.
[0024] A second aspect of this application provides a fault node removal device, comprising:
[0025] The information receiving unit is used to receive abnormal information of the currently faulty replica reported by the client or the primary replica;
[0026] The detection unit is configured to send a detection command for the currently faulty replica to the normal replica corresponding to the currently faulty replica, so as to trigger the normal replica corresponding to the currently faulty replica to detect the currently faulty replica; wherein, the normal replica corresponding to the currently faulty replica includes multiple replicas in the replica group to which the currently faulty replica belongs, excluding the currently faulty replica.
[0027] The result receiving unit is used to receive the detection results fed back by the normal replica corresponding to the current faulty replica;
[0028] The judgment unit is used to determine whether the detection results fed back by the normal replica corresponding to the current faulty replica all indicate that the current faulty replica is abnormal;
[0029] The removal unit is used to send a removal instruction to the current faulty replica to remove the current faulty replica if the detection results fed back by the normal replica corresponding to the current faulty replica indicate that the current faulty replica is abnormal.
[0030] Optionally, in the above-described fault node removal device, the current faulty copy is the primary copy, and the information receiving unit includes:
[0031] The first information receiving unit is used to receive the abnormal information of the primary replica reported by the client; wherein the abnormal information of the primary replica is generated and fed back to the client when the primary replica fails to write data successfully during the process of responding to the data processing request sent to it by the client.
[0032] Optionally, in the above-described fault node removal device, the current faulty replica is any slave replica, and the information receiving unit includes:
[0033] The second information receiving unit is used to receive the abnormal information of the slave replica reported by the primary replica; wherein the abnormal information of the slave replica is generated by the slave replica when it fails to write the data synchronized from the primary replica to the slave replica and is fed back to the primary replica; after responding to the data processing request sent to it by the client and successfully writing the data, the primary replica synchronizes the data to each slave replica in its replica group.
[0034] Optionally, the above-mentioned fault node removal device further includes:
[0035] The lookup unit is used to look up the routing information of the normal replica corresponding to the current faulty replica from the routing information table based on the abnormal information of the current faulty replica.
[0036] The detection unit includes:
[0037] The detection subunit is used to send a detection command for the current faulty replica to each normal replica corresponding to the current faulty replica, based on the routing information of each normal replica corresponding to the current faulty replica.
[0038] Optionally, the above-mentioned fault node removal device further includes:
[0039] The monitoring unit is used to monitor in real time whether the successful removal information from the current faulty copy is received.
[0040] The update unit is used to send the latest copy information to the client when it detects that the successful removal information of the current faulty copy has been received, so as to trigger the client to update the recorded copy information using the latest copy information.
[0041] A third aspect of this application provides an electronic device, comprising:
[0042] Memory and processor;
[0043] The memory is used to store programs;
[0044] The processor is used to execute the program, which, when executed, is specifically used to implement the fault node removal method as described in any of the above.
[0045] The fourth aspect of this application provides a computer storage medium for storing a computer program, which, when executed, implements the fault node removal method as described in any of the preceding claims.
[0046] This application provides a method for removing faulty nodes. During each data processing iteration, the client communicates with the primary replica, and the primary replica communicates with each secondary replica. Therefore, when a replica fails, the client or primary replica obtains and reports the anomaly information. Accordingly, the method receives the anomaly information of the currently faulty replica reported by the client or primary replica, and then sends a probe command to the corresponding normal replica to trigger it to probe the faulty replica. The normal replicas corresponding to the faulty replica include multiple replicas in the replica group to which the faulty replica belongs, excluding the faulty replica itself. Then, the method receives the probe results from the corresponding normal replicas and determines whether all the probe results indicate an anomaly in the faulty replica. If all the probe results from the corresponding normal replicas indicate an anomaly, a removal command is sent to the faulty replica to remove it. This allows for timely detection of potential copy failures, and upon further confirmation of the failure, the faulty copy can be removed without requiring manual intervention. This effectively ensures the timely removal of faulty copies and avoids impacting subsequent data processing requests. Attached Figure Description
[0047] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0048] Figure 1 A flowchart of a fault node removal method provided in an embodiment of this application;
[0049] Figure 2 A flowchart illustrating an example of a method for removing a master copy, provided in an embodiment of this application;
[0050] Figure 3 A flowchart illustrating an example of a method for removing copies provided in this application embodiment;
[0051] Figure 4 A schematic diagram of the architecture of a fault node removal device provided in an embodiment of this application;
[0052] Figure 5 This is a schematic diagram of the architecture of an electronic device provided in an embodiment of this application. Detailed Implementation
[0053] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0054] In this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0055] This application provides a method for removing faulty nodes to address the shortcomings of existing removal methods, which cannot guarantee the timely removal of faulty replicas and avoid affecting response data processing requests.
[0056] Optionally, the faulty node removal method provided in this application embodiment can be specifically applied to the management node in the storage pool, i.e., the Master node in the storage pool. Here, the storage pool is the smallest physical operating cluster unit of the cloud disk.
[0057] like Figure 1 As shown in the figure, an embodiment of this application provides a method for eliminating faulty nodes, which includes the following steps:
[0058] S101. Receive abnormal information about the currently faulty replica reported by the client or the primary replica.
[0059] It's important to note that in a multi-replica mechanism, when data processing is required, the client sends a data processing request to the primary replica in the replica group. This request can be either an input or output request. The primary replica first writes the data, and after successfully writing the data, it synchronizes the data to each secondary replica, allowing each secondary replica to write the data. Therefore, during a single data processing operation, the client communicates with the primary replica, and the primary replica communicates with each secondary replica. This allows the client to obtain the health status of the primary replica, while the primary replica can obtain the monitoring status of the secondary replicas.
[0060] Therefore, in this embodiment, after the client initiates a data processing request, the client obtains the health status of the primary replica, and the primary replica obtains the health status of each secondary replica. Specifically, when the primary replica fails, the client reports the primary replica's error information. When any secondary replica fails, the primary replica reports the error information of the failed secondary replica.
[0061] Optionally, the anomaly information may include the IP information of the currently faulty replica, the ID information of the partition that caused the anomaly, and the ID information of the corresponding physical block.
[0062] S102. Send a probe command for the current faulty replica to the normal replica corresponding to the current faulty replica, so as to trigger the normal replica corresponding to the current faulty replica to probe the current faulty replica.
[0063] Among them, the normal replicas corresponding to the currently faulty replica include multiple replicas in the replica group to which the currently faulty replica belongs, excluding the currently faulty replica.
[0064] Optionally, since subsequent detection results from the normal replicas corresponding to the currently faulty replica are needed to determine whether the currently faulty replica truly has a fault, and thus whether it should be removed, to ensure the accuracy of the results, all replicas in the replica group to which the currently faulty replica belongs, excluding the currently faulty replica itself, are typically considered as the normal replicas corresponding to the currently faulty replica. Of course, if the number of replicas in the replica group to which the currently faulty replica belongs is large, only a portion of the replicas can be selected as the normal replicas corresponding to the currently faulty replica.
[0065] It should be noted that in this embodiment, when a replica fails to process data, it generates an error message and is temporarily designated as a faulty replica. However, the failure to process data may be due to other reasons such as network anomalies, and therefore does not necessarily indicate a fault. Further confirmation is required. Therefore, in this embodiment, it is also necessary to send a probe command for the currently faulty replica to the corresponding normal replica, triggering the normal replica to send a ping probe command to the currently faulty replica, thereby probing the status of the currently faulty replica.
[0066] Optionally, in another embodiment of this application, before sending the probe command for the currently faulty copy to the normal copy corresponding to the currently faulty copy, the following may be further performed:
[0067] Based on the anomaly information of the current faulty replica, retrieve the routing information of the normal replica corresponding to the current faulty replica from the routing information table.
[0068] It's important to note that since a storage pool typically contains multiple replica groups, the management node stores routing information for each replica group. Therefore, based on the anomaly information of the currently failed replica, its routing group can be found in the routing table, and the routing information for its corresponding multiple healthy replicas can be extracted from it. Specifically, the routing information for the healthy replicas corresponding to the currently failed replica can be found based on the partition ID information of the partition that experienced the anomaly and the corresponding physical block ID information.
[0069] Accordingly, in one specific implementation of step S102 in this application embodiment, it includes:
[0070] Based on the routing information of each normal replica corresponding to the current faulty replica, a probe command for the current faulty replica is sent to each normal replica corresponding to the current faulty replica.
[0071] S103. Receive the detection results fed back by the normal replica corresponding to the current faulty replica.
[0072] Specifically, the normal replicas corresponding to the currently faulty replica report the detection results to the management node based on the detection of the currently faulty replica. At this time, the management node receives the detection results reported by each normal replica corresponding to the currently faulty replica.
[0073] S104. Determine whether the detection results fed back by the normal replica corresponding to the current faulty replica all indicate that the current faulty replica is abnormal.
[0074] If it is determined that the detection results fed back by the normal replica corresponding to the current faulty replica all indicate that the current faulty replica is abnormal, then step S105 is executed.
[0075] S105. Send a removal command to the currently faulty copy to remove the currently faulty copy.
[0076] Optionally, the current replicas mainly exist in three operating states: normal, down, and dead. Replicas in the down state no longer provide I / O services, and if a replica cannot recover after a specified time, its state is set to dead. Data from replicas in the dead state is migrated to other normal nodes. Therefore, in this embodiment, a culling instruction can be sent to the currently faulty replica to set its state to dead.
[0077] Optionally, in order to promptly restore the normal operation of the replica group to which the currently faulty replica belongs, in another embodiment of this application, after executing step S105, the following can be further performed:
[0078] The system monitors in real time whether it has received a successful removal message from the current faulty replica. When it detects that a successful removal message has been received from the current faulty replica, it sends the latest replica information to the client to trigger the client to update the recorded replica information using the latest replica information.
[0079] Specifically, after receiving a removal instruction, the data of the currently faulty replica will be migrated to another normal node, which will replace the current faulty replica. Therefore, the relevant information of the node can be obtained as the latest replica information, and the latest replica information can be used to replace the information of the currently faulty replica originally recorded in the client. This updates the replica information of the client, allowing the client to continue to work normally using the new replica information.
[0080] This application provides a method for removing faulty nodes. During each data processing iteration, the client communicates with the primary replica, and the primary replica communicates with each secondary replica. Therefore, when a replica fails, the client or primary replica obtains and reports the anomaly information. Accordingly, the method receives the anomaly information of the currently faulty replica reported by the client or primary replica, and then sends a probe command for the currently faulty replica to the corresponding normal replica, triggering the normal replica to probe the currently faulty replica. The normal replicas corresponding to the currently faulty replica include multiple replicas in the replica group to which the currently faulty replica belongs, excluding the currently faulty replica itself. Then, the method receives the probe results from the normal replicas corresponding to the currently faulty replica and determines whether all the probe results indicate an anomaly in the currently faulty replica. If it is determined that all the probe results from the normal replicas corresponding to the currently faulty replica indicate an anomaly, a removal command is sent to the currently faulty replica to remove it. This allows for timely detection of potential copy failures, and upon further confirmation of the failure, the faulty copy can be removed without requiring manual intervention. This effectively ensures the timely removal of faulty copies and avoids impacting subsequent data processing requests.
[0081] Optionally, in another embodiment of this application, when the current faulty copy is the primary copy, one specific implementation of step S101 includes:
[0082] Receive abnormal information about the primary replica reported by the client.
[0083] Specifically, the primary replica generates and sends error messages to the client when it fails to write data during a data processing request from the client. Upon receiving these error messages, the client then reports them to the management node.
[0084] Therefore, when the current faulty replica is the primary replica, i.e., when the primary replica fails, taking a three-replica mechanism as an example, the method for removing the primary replica is as follows: Figure 2 As shown, the specific steps include:
[0085] 1. The client initiates an IO request and uses the cloud disk's hash algorithm to find the corresponding primary replica for data processing.
[0086] 2. If the primary replica fails to process data, it will send an error message to the client.
[0087] 3. After receiving the exception information from the primary replica, the client reports the exception information to the Master node in the storage pool.
[0088] 4. After receiving the abnormal information of the primary replica, the Master node finds the replica information of the replica group to which it belongs based on the abnormal information, and sends probe commands to the two secondary replicas of the replica group based on the replica information.
[0089] 5. Both replicas simultaneously send ping probe commands to the primary replica.
[0090] 6. The two replicas report the abnormal information detected to the Master node.
[0091] 7. After receiving two reports of primary replica anomalies from the secondary replicas, the Master node issues a removal command to the primary replica.
[0092] 8. After the primary replica is successfully removed, a message indicating successful removal is sent to the Master node.
[0093] 9. The Master node updates the latest replica information to the client.
[0094] Optionally, in another embodiment of this application, when the current faulty copy is a slave copy, one specific implementation of step S101 includes:
[0095] Receive exception information from the slave replica reported by the primary replica.
[0096] The exception information for a slave replica is generated and reported back to the master replica when the slave replica fails to successfully write data synchronized from the master replica. After responding to a data processing request from a client and successfully writing the data, the master replica synchronizes the data to every slave replica in its replica group. That is, after receiving a data processing request, the master replica attempts to write the data and, after successful writing, synchronizes the data to the slave replicas. When a slave replica fails to write data, it indicates an exception has occurred and will report the exception information to the master replica. The master replica then reports the exception information to the management node.
[0097] Therefore, when the currently faulty replica is a slave replica, that is, when any slave replica fails, taking a three-replica mechanism as an example, the method for removing the slave replica is as follows: Figure 3 As shown, the specific steps include:
[0098] 1. The client initiates an IO request and uses the cloud disk's hash algorithm to find the corresponding primary replica for data processing.
[0099] 2. After successfully processing the data, the primary replica will synchronize the written data to the two secondary replicas in the replica group shown.
[0100] 3. If an exception occurs in replica 1 of the two replicas, its data writing fails and it sends the exception information back to the primary replica. Replica 2, on the other hand, successfully writes the data and returns a success message.
[0101] 4. The primary replica will report the abnormal information from replica 1 to the Master node in the storage pool.
[0102] 5. After receiving the abnormal information from replica 1, the Master node finds the replica information of the replica group to which it belongs based on the abnormal information, and sends probe commands to the primary replica and replica 2 of the replica group based on the replica information.
[0103] 6. The primary replica and the secondary replica 2 simultaneously send ping probe commands to the secondary replica 1.
[0104] 7. The primary replica and secondary replica 2 report the abnormal information of secondary replica 1 to the Master node based on the detected abnormal information.
[0105] 8. After receiving the information from the two replicas that replica 2 is abnormal, the Master node issues a removal command to replica 2.
[0106] 9. After successfully removing a replica from replica 2, send a message to the Master node indicating successful removal.
[0107] 10. The Master node updates the latest replica information to the client.
[0108] Another embodiment of this application provides a fault node removal device, such as... Figure 4 As shown, it includes:
[0109] The information receiving unit 401 is used to receive abnormal information of the currently faulty replica reported by the client or the primary replica.
[0110] The detection unit 402 is used to send a detection command of the current faulty copy to the normal copy corresponding to the current faulty copy, so as to trigger the normal copy corresponding to the current faulty copy to detect the current faulty copy.
[0111] Among them, the normal replicas corresponding to the currently faulty replica include multiple replicas in the replica group to which the currently faulty replica belongs, excluding the currently faulty replica.
[0112] The result receiving unit 403 is used to receive the detection results fed back by the normal replica corresponding to the current faulty replica.
[0113] The judgment unit 404 is used to determine whether the detection results fed back by the normal replica corresponding to the current faulty replica all indicate that the current faulty replica is abnormal.
[0114] The removal unit 405 is used to send a removal command to the current faulty copy to remove the current faulty copy if the detection results fed back by the normal copy corresponding to the current faulty copy indicate that the current faulty copy is abnormal.
[0115] Optionally, in another embodiment of the fault node removal device provided in this application, the current fault copy is the primary copy, and the information receiving unit includes:
[0116] The first information receiving unit is used to receive abnormal information about the primary replica reported by the client.
[0117] The exception information of the primary replica is generated and fed back to the client when the primary replica fails to write data during the process of responding to the data processing request sent to it by the client.
[0118] Optionally, in another embodiment of the fault node removal device provided in this application, the current faulty copy is any slave copy, and the information receiving unit includes:
[0119] The second information receiving unit is used to receive abnormal information from the slave replica reported by the master replica.
[0120] Specifically, exception information for a replica is generated and reported back to the primary replica when the replica fails to write data synchronized from the primary replica. After responding to a data processing request from a client and successfully writing the data, the primary replica synchronizes the data to every replica in its replica group.
[0121] Optionally, in another embodiment of the fault node removal device provided in this application, the device further includes:
[0122] The lookup unit is used to retrieve the routing information of the normal replica corresponding to the current faulty replica from the routing information table based on the anomaly information of the current faulty replica.
[0123] In this embodiment of the application, the detection unit includes:
[0124] The probe subunit is used to send probe commands for the current faulty replica to each normal replica corresponding to the current faulty replica, based on the routing information of each normal replica corresponding to the current faulty replica.
[0125] Optionally, in another embodiment of the fault node removal device provided in this application, the device further includes:
[0126] The monitoring unit is used to monitor in real time whether the successful removal information of the current faulty copy has been received.
[0127] The update unit is used to send the latest replica information to the client when it detects that the successful removal information of the current faulty replica has been received, so as to trigger the client to update the recorded replica information with the latest replica information.
[0128] It should be noted that the specific working process of each unit provided in the above embodiments of this application can be referred to the implementation process of the corresponding steps in the above method embodiments, and will not be repeated here.
[0129] Another embodiment of this application provides an electronic device, such as... Figure 5 As shown, it includes:
[0130] Memory 501 and processor 502.
[0131] The memory 501 is used to store the program.
[0132] The processor 502 is used to execute the program stored in the memory 501. When the program is executed, it is specifically used to implement the fault node removal method provided in any of the above embodiments.
[0133] Another embodiment of this application provides a computer storage medium for storing a computer program, which, when executed, implements the fault node removal method provided in any of the above embodiments.
[0134] It should be noted that the computer storage media provided in the embodiments of this application include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient media, such as modulated data signals and carrier waves.
[0135] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0136] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for eliminating faulty nodes, characterized in that, Management nodes used in the storage pool include: Receive abnormal information of the currently faulty replica reported by the client or the primary replica. The abnormal information includes the IP information of the currently faulty replica, the ID information of the small table where the abnormality occurred, and the ID information of the corresponding physical block. Based on the anomaly information of the current faulty replica, the routing information of the normal replica corresponding to the current faulty replica is retrieved from the routing information table; Based on the routing information of each normal replica corresponding to the current faulty replica, a probe command for the current faulty replica is sent to each normal replica corresponding to the current faulty replica, so as to trigger the normal replica corresponding to the current faulty replica to probe the current faulty replica; wherein, the normal replica corresponding to the current faulty replica includes multiple replicas in the replica group to which the current faulty replica belongs, excluding the current faulty replica. Receive the detection results fed back by the normal replica corresponding to the currently faulty replica; Determine whether the detection results fed back by the normal replica corresponding to the current faulty replica all indicate that the current faulty replica is abnormal; If it is determined that the detection results fed back by the normal replica corresponding to the current faulty replica all indicate that the current faulty replica is abnormal, then a removal instruction is sent to the current faulty replica to remove the current faulty replica.
2. The method according to claim 1, characterized in that, The currently faulty replica is the primary replica, and the abnormal information of the currently faulty replica reported by the receiving client or the primary replica includes: The system receives exception information about the primary replica reported by the client; wherein the exception information about the primary replica is generated and fed back to the client when the primary replica fails to write data during the process of responding to a data processing request sent to it by the client.
3. The method according to claim 1, characterized in that, The currently faulty replica can be any slave replica, and the abnormal information of the currently faulty replica reported by the receiving client or the primary replica includes: The system receives exception information from the slave replicas reported by the primary replica; wherein the exception information from the slave replicas is generated by the slave replicas when they are unable to successfully write data synchronized from the primary replicas to the slave replicas and is fed back to the primary replicas; after responding to a data processing request sent to it by the client and successfully writing the data, the primary replicas synchronize the data to each slave replica in its replica group.
4. The method according to claim 1, characterized in that, After sending the culling instruction to the currently faulty replica, the method further includes: Real-time monitoring to see if successful removal information is received from the currently faulty replica; When it is detected that the successful removal information of the current faulty copy has been received, the latest copy information is sent to the client to trigger the client to update the recorded copy information using the latest copy information.
5. A fault node removal device, characterized in that, Management nodes used in the storage pool include: The information receiving unit is used to receive abnormal information of the currently faulty replica reported by the client or the primary replica. The abnormal information includes the IP information of the currently faulty replica, the ID information of the small table where the abnormality occurred, and the ID information of the corresponding physical block. The lookup unit is used to look up the routing information of the normal replica corresponding to the current faulty replica from the routing information table based on the abnormal information of the current faulty replica. The detection unit is configured to send a detection command for the current faulty replica to each normal replica corresponding to the current faulty replica based on the routing information of each normal replica corresponding to the current faulty replica, so as to trigger the normal replica corresponding to the current faulty replica to detect the current faulty replica; wherein, the normal replica corresponding to the current faulty replica includes multiple replicas other than the current faulty replica in the replica group to which the current faulty replica belongs. The result receiving unit is used to receive the detection results fed back by the normal replica corresponding to the current faulty replica; The judgment unit is used to determine whether the detection results fed back by the normal replica corresponding to the current faulty replica all indicate that the current faulty replica is abnormal; The removal unit is used to send a removal instruction to the current faulty replica to remove the current faulty replica if the detection results fed back by the normal replica corresponding to the current faulty replica indicate that the current faulty replica is abnormal.
6. The apparatus according to claim 5, characterized in that, The currently faulty replica is the primary replica, and the information receiving unit includes: The first information receiving unit is used to receive the abnormal information of the primary replica reported by the client; wherein the abnormal information of the primary replica is generated and fed back to the client when the primary replica fails to write data successfully during the process of responding to the data processing request sent to it by the client.
7. The apparatus according to claim 5, characterized in that, The currently faulty replica is any slave replica, and the information receiving unit includes: The second information receiving unit is used to receive the abnormal information of the slave replica reported by the primary replica; wherein the abnormal information of the slave replica is generated by the slave replica when it fails to write the data synchronized from the primary replica to the slave replica and is fed back to the primary replica; after responding to the data processing request sent to it by the client and successfully writing the data, the primary replica synchronizes the data to each slave replica in its replica group.
8. An electronic device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is used to execute the program, which, when executed, is specifically used to implement the fault node removal method as described in any one of claims 1 to 4.
9. A computer storage medium, characterized in that, Used to store a computer program, which, when executed, is used to implement the fault node removal method as described in any one of claims 1 to 4.