Distributed storage node fault detection and repair device and method

Abstract

The invention relates to the technical field of distributed storage, and discloses a distributed storage node fault detection and repair device and method, and the device comprises a detection module which is used for regularly collecting and reporting node state information; the analysis module is used for receiving and analyzing the node state information so as to identify a fault and generate a fault analysis report containing a fault level; the repairing module is used for executing a local strategy or a global repairing strategy corresponding to the fault level according to the fault analysis report; the storage module is used for storing system configuration, operation data and user service data in an encryption mode; the alarm module is connected with the analysis module and is used for triggering graded alarm according to the fault grade of the fault analysis report; and the communication module is connected to each module in the device and is used for data transmission and instruction issuing between the modules and between the nodes. The operation and maintenance automation level and reliability of the distributed storage system are fundamentally improved through the modular and distributed collaborative architecture design.

Description

Technical Field

[0001] This application relates to the field of distributed storage technology, and for example to an apparatus and method for detecting and repairing faults in distributed storage nodes. Background Technology

[0002] With the rapid development of cloud computing and big data technologies, enterprises' demand for large-scale data storage continues to grow, making distributed storage technology mainstream. It integrates the distributed disk space of multiple machines through a network to form virtual storage devices, with data distributed across various storage nodes. Nodes collaborate to achieve high scalability and high reliability. However, increasing the number of nodes increases the probability of hardware failures, network outages, and data corruption. If these issues are not addressed promptly, they will affect data integrity and system availability. Therefore, efficient fault detection and recovery methods for nodes will greatly improve the reliability and stability of distributed storage systems.

[0003] Currently, fault detection and recovery of distributed storage nodes mainly rely on two methods: manual inspection and single-node alarms. Manual inspection requires maintenance personnel to check all storage nodes one by one according to a fixed cycle, judging whether the nodes are abnormal by checking hardware status and log information. Single-node alarms are implemented by each storage node independently monitoring its own status and triggering local alarm signals when an anomaly is detected, such as an indicator light illuminating or a simple log entry. However, these two methods have significant technical problems: First, detection is not timely. The fixed cycle of manual inspection cannot cope with sudden node failures, and single-node alarms can only passively report local anomalies. Neither can capture changes in node status in real time, resulting in a significant delay in fault detection. Second, fault handling is lagging. Regardless of the type of fault, maintenance personnel must manually intervene to analyze the fault type and then formulate and implement a repair plan. The entire process is time-consuming and significantly increases system downtime. Third, there is a high risk to data. If a faulty node is not repaired in time, it may lead to insufficient data replicas, and in severe cases, it may cause permanent data loss or interruption of read and write services, threatening data security and system availability. Therefore, there is an urgent need for a distributed storage node fault detection and repair method to solve the above problems.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0006] This disclosure provides an apparatus and method for detecting and repairing faults in distributed storage nodes, in order to solve the technical problems of high detection latency, difficulty in fault location, and low repair efficiency in existing distributed storage node fault detection and recovery methods.

[0007] In some embodiments, a distributed storage node fault detection and repair apparatus includes: The detection module is deployed on the management node and each storage node in the distributed storage system. It is used to collect node status information periodically and report the node status information to the analysis module. The analysis module, deployed on the management node, is used to receive and analyze node status information to identify faults and generate fault analysis reports containing fault levels. The repair module is deployed on the management node and each storage node, and is used to execute local or global repair strategies corresponding to the fault level based on the fault analysis report. The storage module is deployed on the management node and each storage node to store system configuration, runtime data and user business data in an encrypted manner. The alarm module is deployed on the management node and each storage node, and is connected to the analysis module. It is used to trigger graded alarms based on the fault level in the fault analysis report. The communication module is deployed on the management node, each storage node, and the client node. It connects the detection module, analysis module, repair module, storage module, and alarm module, and is used for data transmission and command issuance between modules and nodes.

[0008] In some embodiments, the detection module includes: The local detection unit is deployed on each storage node to periodically collect the node status information of the corresponding storage node at a first configurable period. The node status information includes: hardware status indicators, storage status indicators and network status indicators. The global detection unit, deployed on the management node, is used to periodically summarize and analyze the node status information of all storage nodes at a second configurable period to perform inter-node collaborative status detection.

[0009] In some embodiments, hardware status indicators include CPU utilization, memory usage, disk I / O rate, and number of bad sectors on the disk; storage status indicators include data copy integrity determined based on check hash values ​​and storage capacity utilization; and network status indicators include inter-node communication latency and packet loss rate.

[0010] In some embodiments, during the generation of a fault analysis report, the analysis module is configured as follows: The received node status information is compared with a preset fault threshold. The fault level is determined based on the rules pre-set in the fault rule base, where the fault level includes minor fault, moderate fault and severe fault. Based on the fault level and the corresponding node status information, the fault is located and a fault analysis report is generated. The fault analysis report includes the fault node identifier, the specific faulty component, and the scope of impact, which is used by the repair module to determine the corresponding repair strategy. Historical fault data is batch-verified at preset intervals, and the fault rule base is improved based on the verification results.

[0011] In some embodiments, the repair module includes: The local repair unit is deployed on each storage node to execute local repair strategies corresponding to minor faults in the fault analysis report. These local repair strategies include shutting down unnecessary processes, adjusting data read / write priorities, deleting redundant logs, or migrating cold data to low-load nodes. The global repair unit, deployed on the management node, is used to execute global repair strategies corresponding to moderate or severe faults in the fault analysis report. These global repair strategies include copying data from healthy nodes to rebuild damaged copies, migrating storage tasks from offline nodes to standby nodes, or triggering hardware replacement alarms and rebuilding data from other copies.

[0012] In some embodiments, the repair module has a built-in repair strategy library, which is associated with the local repair unit and the global repair unit respectively. The repair strategy library is used to automatically match the local repair strategy or global repair strategy corresponding to the fault level according to the fault analysis report, so that the local repair unit or the global repair unit can execute it. The repair module is also configured to provide real-time progress feedback during the repair process and generate a repair acceptance report to be sent back to the analysis module after the repair is completed.

[0013] In some embodiments, the storage module includes: Node storage units are deployed on each storage node and are used to store user business data, local detection historical data, and data copy information using a distributed hash table; The management storage unit, deployed on the management node, is used to store system configuration data, global fault logs, and node topology information; The storage module uses AES-256 encryption to store data and transmits data via SSL protocol during data copy synchronization.

[0014] In some embodiments, the communication module is configured as follows: The node status information collected by the detection module is transmitted using the UDP protocol to reduce latency; The repair instructions issued by the analysis module and the repair acceptance report returned by the repair module are transmitted using the TCP protocol to ensure reliability.

[0015] In some embodiments, the alarm module includes: The local alarm unit is deployed on each storage node and connected to the detection module. It is used to immediately trigger a local audible and visual alarm when the detection module detects an emergency fault. The centralized alarm unit, deployed on the management node and connected to the analysis module, is used to perform tiered push notifications based on the fault level, including: for minor faults, only recording and pushing system logs; for moderate faults, pushing electronic message alarms to the operation and maintenance terminal; and for severe faults, pushing electronic messages and initiating voice communication alarms to the operation and maintenance personnel. After the fault is repaired, the centralized alarm unit automatically sends an alarm cancellation notification, forming an alarm closed loop.

[0016] In some embodiments, a method for detecting and repairing faults in distributed storage nodes, wherein the apparatus for detecting and repairing faults in distributed storage nodes described above includes: Periodically trigger the status detection of each storage node in the distributed storage system, and collect node status information including hardware, storage and network status; The node status information is compared with the preset fault threshold, and the fault level is determined based on the rules preset in the fault rule base. The fault levels include minor fault, moderate fault and severe fault. Based on the fault level and corresponding node status information, fault location is performed, and a fault analysis report is generated that includes the fault node identifier, specific faulty component, and scope of impact. Perform corresponding repair operations based on the fault level in the fault analysis report, and provide real-time feedback on the repair progress during the repair process. For minor faults, execute local repair strategies at the fault node; for moderate or severe faults, coordinate multiple nodes to execute global repair strategies. If the repair is successful, a repair completion confirmation message will be generated and an alarm cancellation notification will be sent. If the repair fails, the alarm will be escalated to notify manual intervention. After processing the repair results, a timed trigger detection is initiated to form a closed loop of fault detection and repair.

[0017] The apparatus and method for distributed storage node fault detection and repair provided in this disclosure can achieve the following technical effects: In this application, the detection module is deployed on the management node and each storage node. It can periodically collect the status information of all nodes and actively report it to the analysis module, realizing the timed acquisition of node status information, overcoming the lag of manual inspection, and significantly reducing detection latency. The analysis module is centrally deployed on the management node. It can receive and integrate the status information of all nodes, identify faults through standardized analysis, and generate fault analysis reports with fault levels. This avoids the tedious process of requiring maintenance personnel to manually analyze fault types. Through centralized and accurate fault identification and level labeling, it solves the problem of fault location relying on manual methods and difficulty in quickly and accurately judging faults. The repair module is distributed on the management node and each storage node. It can execute corresponding local or global repair strategies based on the fault level in the fault analysis report, realizing automated response from local optimization to global data reconstruction, greatly reducing manual intervention and system downtime, thereby improving repair efficiency. At the same time, the communication module connects the management node, storage nodes, and client nodes, ensuring the timely transmission of detection data, fault analysis reports, and repair instructions, providing communication support for efficient collaboration among modules. The storage module provides a secure data foundation for the entire process. The storage module stores system configuration, operational data, and business data in an encrypted manner, ensuring data security during fault handling and avoiding the risk of data loss during fault repair. The alarm module triggers tiered alarms based on fault severity, assisting maintenance personnel in focusing on critical faults and further improving the fault handling closed loop. This application, through a modular and distributed collaborative architecture design, fundamentally improves the automation level and reliability of distributed storage system operations and maintenance.

[0018] The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description

[0019] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are shown as similar elements. The drawings are not to be scaled. And wherein: Figure 1 This is a system block diagram of a distributed storage node fault detection and repair device provided in an embodiment of this disclosure; Figure 2 This is a timing diagram of data interaction between modules in a distributed storage node fault detection and repair device provided in an embodiment of this disclosure; Figure 3 This is a schematic diagram of a fault classification process provided in an embodiment of this disclosure; Figure 4 This is an execution flowchart of a distributed storage node fault detection and repair device provided in an embodiment of this disclosure; Figure 5This is a flowchart of a method for fault detection and repair of a distributed storage node provided in an embodiment of this disclosure. Detailed Implementation

[0020] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0021] The terms "first," "second," etc., used in the embodiments of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0022] Unless otherwise stated, the term "multiple" means two or more.

[0023] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0024] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0025] The term "correspondence" can refer to an association or binding relationship. The correspondence between A and B means that there is an association or binding relationship between A and B.

[0026] Distributed storage is a technology that integrates the disk space of various machines in an enterprise into virtual storage devices through a network, distributing data across multiple independent devices (each device being a storage node). However, increasing the number of nodes increases the probability of hardware failures, network outages, and data corruption. Efficient node fault detection and recovery are crucial for system reliability and stability. Currently, fault detection and recovery mainly rely on manual inspections and single-node alarms, but these two methods have significant technical problems: First, detection is not timely. Manual inspections, with their fixed cycles, are difficult to handle sudden failures, and single-node alarms only passively report local anomalies, neither of which can capture real-time changes in node status, resulting in significant delays in fault detection. Second, fault handling is lagging. All faults require manual intervention from maintenance personnel to analyze the type, formulate, and execute repair plans, which is time-consuming and increases system downtime. Third, data risk is high. Failure to repair faulty nodes in a timely manner may lead to insufficient data replicas, and in severe cases, permanent data loss or interruption of read and write services, threatening data security and system availability.

[0027] To address the aforementioned problems, this disclosure provides an apparatus for detecting and repairing faults in distributed storage nodes. The apparatus for detecting and repairing faults in distributed storage nodes provided in this disclosure will be described in detail below with reference to the accompanying drawings.

[0028] Figure 1 This is a system block diagram of a distributed storage node fault detection and repair device provided in an embodiment of this disclosure. Figure 2 This is a timing diagram of data interaction between modules in a distributed storage node fault detection and repair device provided in this embodiment of the disclosure. Combined with... Figure 1 and Figure 2 The device includes a detection module, an analysis module, a repair module, a storage module, an alarm module, and a communication module. The following is a detailed explanation of each module and the cooperation between them.

[0029] In some embodiments, the detection module is the core component for fault perception of distributed storage nodes. It is deployed on the management node and each storage node of the distributed storage system to periodically collect node status information of the entire system and report complete and accurate status data to the analysis module, providing basic data support for subsequent fault identification and location. The detection module includes two collaborative functional units: a local detection unit and a global detection unit.

[0030] In some embodiments, the local detection unit is deployed on each storage node and can perform timed data collection according to a first configurable period. This first configurable period can be flexibly set according to system scale and detection accuracy requirements, for example, once per minute. The collected data is the node status information of the storage node, which includes three main categories: hardware status indicators, storage status indicators, and network status indicators. Hardware status indicators include CPU utilization, memory usage, disk I / O rate, and the number of bad sectors on the disk. These indicators directly reflect the operational health of the storage node hardware. Storage status indicators include data replica integrity and storage capacity utilization. Data replica integrity is determined by verifying hash values, i.e., by comparing the actual hash value of the data with a preset hash value to confirm whether the replica is corrupted. Network status indicators include inter-node communication latency and packet loss rate, which can effectively monitor the stability of data interaction between nodes.

[0031] In some embodiments, the global detection unit is deployed on the management node and operates according to a second configurable cycle, which can also be adjusted according to actual needs, for example, it can be set to once every 5 minutes. Its core function is to aggregate the node status information reported by all storage nodes through the local detection unit and perform secondary analysis on this information, focusing on inter-node collaborative status detection. Inter-node collaborative status detection specifically includes two aspects: first, verifying the data replica synchronization progress of each storage node to confirm whether the data replicas across nodes are consistent; second, assessing the load balancing of the entire system to determine whether there is a situation where some nodes are overloaded while other nodes are idle. The setting of the global detection unit can effectively avoid false alarms caused by local hardware fluctuations or temporary network interference of a single storage node. For example, a short-term network delay in a single node will not be directly judged as a fault. At the same time, it can also avoid missed alarms. For example, if the data replica of a node is damaged and not identified by its own local detection unit, the anomaly can be found through global data comparison, thereby ensuring that the status information reported to the analysis module has global accuracy and completeness.

[0032] In actual operation, both the local and global detection units can have built-in timer triggers. When their respective set detection cycles arrive, the timer triggers will automatically start the detection process. After each detection is completed, the two units will integrate the collected raw indicator data and collaborative status analysis results to form a node status report. The report will clearly indicate the specific values ​​of each indicator and any anomalies found. The generated node status report will be transmitted via the communication module to the analysis module deployed on the management node, providing complete and timely data input for subsequent fault rule matching and fault level determination.

[0033] In some embodiments, the analysis module receives node status reports transmitted by the detection module through the communication module, and performs systematic and in-depth analysis on the node status information in the reports. This allows for accurate identification of fault types, determination of fault levels, and generation of a fault analysis report containing key decision-making information. This provides a direct and reliable basis for the subsequent repair module to perform targeted repair operations. The specific process is as follows: The analysis module first retrieves the fault threshold parameters preset in the storage module. These parameters are set based on the operating characteristics of the distributed storage system, hardware capacity, and business stability requirements. For example, for hardware status indicators, the thresholds are set as follows: CPU utilization > 90%, disk bad sector number > 5; for network status indicators, communication latency > 100ms; and for storage status indicators, storage capacity utilization > 85%. Then, the actual indicator values ​​collected by the detection module are compared with these preset thresholds one by one to initially screen out abnormal indicators that exceed the normal operating range, laying the foundation for subsequent fault determination.

[0034] After completing threshold comparisons and filtering out abnormal indicators, the analysis module uses rules pre-defined in its fault rule base to accurately match the fault types corresponding to the abnormal indicators and classify the fault levels. Fault levels are categorized into three types: minor, moderate, and severe. The fault rule base predefines the correspondence between combinations of abnormal indicators and fault types. For example, "high CPU utilization on a single node and disk capacity approaching the utilization threshold" corresponds to a minor fault; "mismatched data replica verification hash values ​​and frequent fluctuations in inter-node communication latency" corresponds to a moderate fault; and "three consecutive unresponsive checks and the number of bad sectors on the disk far exceeding the threshold" corresponds to a severe fault (i.e., node offline or physical disk damage). Through this rule-based matching, the analysis module can quickly determine the severity of each anomaly, ensuring reasonable priority allocation for subsequent repair operations.

[0035] While determining the fault level, the analysis module simultaneously performs refined fault localization. Specifically, based on the node identification information of the abnormal indicator, the analysis module determines the specific location of the faulty node (such as the faulty node's IP address); based on the hardware or functional module corresponding to the abnormal indicator, it identifies the specific components involved in the fault (such as the ID of a disk, the port number of a network card); and then, combined with the faulty node's location in the system topology and its business relationships, it assesses the potential scope of the fault (such as whether it affects the read / write services of other nodes, whether it leads to insufficient data replicas, etc.). Subsequently, the analysis module integrates this key information to form a structured fault analysis report. This report directly determines the type of repair strategy to be adopted by the repair module and is the core basis for initiating the repair operation.

[0036] In some embodiments, to continuously improve the accuracy and adaptability of fault identification, the analysis module is also configured to perform batch verification of historical fault data at preset intervals. In practical applications, the preset interval can be set according to the system's operational scale, for example, performing batch verification once per hour. During each verification cycle, the analysis module retrieves archived historical fault logs, historical node status reports, and fault analysis reports from the storage module to review the accuracy of past fault determinations and the applicability of rules in the fault rule base. If it is found that a fault threshold is set too high, leading to multiple missed reports of similar faults, or that the fault type corresponding to a certain rule deviates significantly from the actual situation, resulting in misjudgment, the fault rule base will be dynamically adjusted based on the verification results. For example, the threshold parameters of relevant indicators may be optimized, and new combinations of abnormal indicators and corresponding rules for fault types may be added, thereby achieving self-iteration and continuous optimization of the module's fault identification capabilities.

[0037] In some embodiments, the repair module is deployed on both the management node and each storage node. This leverages the storage nodes' direct control over local resources for rapid local repair, while also utilizing the management node's global resource scheduling capabilities to handle complex cross-node faults. Based on the fault analysis report generated by the analysis module, the repair module accurately matches the fault level and executes the corresponding local or global repair strategy, eliminating the inefficient manual intervention required in traditional fault handling. The repair module comprises two core, collaborative components: a local repair unit and a global repair unit. These two units have clearly defined roles in terms of deployment location, processing objects, and strategy content. Figure 3 This is a schematic diagram of a fault classification process provided in an embodiment of this disclosure. The following is in conjunction with... Figure 3 A detailed introduction will follow.

[0038] In some embodiments, the local repair unit is deployed on each storage node, specifically responsible for handling minor faults identified in the fault analysis report. These faults only affect the local operation of a single storage node, do not require access to other node resources, and can be quickly resolved through localized operations. Specifically, the local repair strategy includes: when a node load is detected to be too high (e.g., CPU utilization exceeds a preset threshold), performing a "shut down unnecessary processes" operation to clear idle or low-priority background tasks to free up hardware resources; when concentrated business read / write requests cause response delays, performing a "adjust data read / write priority" operation to set read / write requests for core business data to high priority, ensuring that critical business is not affected; when the node disk space is close to the capacity threshold due to log accumulation, performing a "delete redundant logs" operation to clean up archived or worthless log files to free up storage space; when storage capacity is consistently high, performing a "migrate cold data to low-load nodes" operation to transfer infrequently accessed cold data to other resource-rich storage nodes to balance local storage pressure. The scope of the local repair strategy is limited to a single storage node, and the execution process does not require interruption of global business operations, enabling fault repair to be completed without the user's awareness.

[0039] In some embodiments, the global repair unit is deployed on the management node and mainly handles the moderate and severe faults identified in the fault analysis report. These two types of faults often affect cross-node data consistency or core hardware functions and require the coordination of resources from multiple nodes to complete the repair. The corresponding global repair strategies include: When data copy corruption is detected (e.g., checksum hash value mismatch), the operation "copy data from healthy nodes to rebuild damaged copies" is executed. The management node will first locate the healthy node storing the same data copy, and then schedule network resources to copy the complete data to the faulty node, ensuring that the number of data copies is restored to the preset safe value; When a node goes offline due to network interruption or hardware failure, the operation "migrate the storage tasks of the offline node to the standby node" is executed. The management node will immediately activate the standby node resources and take over the read and write services of the offline node. After the offline node recovers, the newly added data on the standby node will be synchronized back to the original node to ensure business continuity; When physical disk damage is detected (e.g., the number of bad sectors far exceeds the threshold), the operation "trigger hardware replacement alarm and rebuild data from other copies" is executed. On the one hand, the hardware replacement notification is pushed to the operation and maintenance personnel through the alarm module, and on the other hand, data is copied from other healthy nodes to the newly replaced disk to ensure that data is not lost and storage function is quickly restored.

[0040] In some embodiments, to ensure accurate matching of repair strategies and traceability of the process, the repair module also has a built-in repair strategy library with real-time feedback and report return capabilities. The repair strategy library is associated with both local and global repair units, and is used to automatically match corresponding repair strategies based on the fault level, fault type, and node information in the fault analysis report. For example, when the report shows "The storage capacity utilization rate of a certain storage node exceeds the threshold (minor fault)," the repair strategy library will directly match the "delete redundant logs" or "migrate cold data" strategy for the local repair unit of that node, completing the strategy selection without manual intervention and significantly improving repair efficiency. Simultaneously, during the repair operation, the repair module will provide real-time feedback on the repair progress to the analysis module, such as "Data reconstruction completed 80%," allowing the analysis module to monitor the repair dynamics in real time.

[0041] In some embodiments, once all repair operations are completed, the repair module generates a structured repair acceptance report. The report includes information such as the repair node identifier, fault type, execution strategy, repair time, and result verification, and is transmitted back to the analysis module via the communication module. Upon receiving the report, the analysis module performs secondary verification of the repair results. After confirming that the fault has been resolved, the report is archived to the storage module, ensuring that every fault repair is traceable and verifiable.

[0042] In some embodiments, the storage module is deployed on each storage node and management node of the distributed storage system, satisfying the local data storage needs of the storage nodes while ensuring centralized control of global configuration and logs by the management node. The storage module is divided into node storage units and management storage units: the node storage unit is deployed on each storage node, and its core task is to store data directly associated with that node, including user business data, local detection historical data (such as past hardware status indicators and fault records), and data copy information, and uses distributed hash table (DHT) technology to achieve data sharding storage; the management storage unit is deployed on the management node, responsible for storing the core configuration and global log data of the entire system, specifically including system configuration data (such as the collection cycle of the detection module, the fault threshold of the analysis module, and the strategy parameters of the repair module), global fault logs (including fault analysis reports and repair acceptance reports of all nodes), and node topology information (such as the IP address, hardware model, and network connection relationship of each node). This centralized storage of data provides the foundation for the management node to conduct global detection, analysis, and repair, while also supporting log backtracking and auditing operations. For example, maintenance personnel can query the fault records of a node within 30 days, facilitating root cause analysis and system optimization.

[0043] In some embodiments, during static storage, the storage module encrypts all data (including user business data, configuration data, and log data) using the AES-256 encryption algorithm. This algorithm is an internationally recognized high-strength symmetric encryption standard with a key length of 256 bits, effectively resisting data theft and tampering attacks and ensuring data security on the storage medium. During dynamic transmission, when data replicas are synchronized across different nodes, the storage module transmits data via the SSL protocol. The SSL protocol establishes a secure session in the transmission link, encrypting and authenticating the transmitted data to prevent interception, eavesdropping, or tampering during network transmission, ensuring the reliability of cross-node data transmission in distributed storage scenarios.

[0044] In some embodiments, a communication module is deployed on the management node, each of the storage nodes, and the client node, connecting the detection module, analysis module, repair module, storage module, and alarm module. It is used for data transmission and command issuance between modules and nodes. The communication module is adaptable as needed: for node status information collected by the detection module, UDP protocol is used for transmission. UDP is a connectionless protocol, eliminating the need for complex connection sessions and resulting in extremely low data transmission latency. Node status information needs to be reported to the analysis module in real time to ensure the timeliness of fault detection (e.g., local detection data collected every minute needs to be quickly aggregated). The low latency of UDP protocol perfectly meets this requirement, ensuring that the analysis module can obtain the operating status of all system nodes in a timely manner. For repair commands issued by the analysis module (e.g., sending minor fault repair commands to storage nodes) and repair acceptance reports fed back by the repair module, TCP protocol is used for transmission. TCP is a connection-oriented reliable transmission protocol with mechanisms such as data retransmission, flow control, and error checking, ensuring that commands and reports are not lost, duplicated, or transmitted in the correct order. Repair instructions directly determine the direction of fault handling, and repair acceptance reports are the key basis for the analysis module to confirm that the fault has been resolved. Both require absolutely reliable transmission guarantees. The characteristics of the TCP protocol can fully meet this requirement, avoiding interruption of fault repair or misjudgment of results due to transmission anomalies.

[0045] In some embodiments, the communication module also has the ability to dynamically adjust bandwidth. During peak fault periods (such as when multiple nodes fail simultaneously and a large amount of detection data and repair instructions need to be transmitted at the same time), it will automatically prioritize the allocation of bandwidth to the transmission of repair instructions and acceptance reports, ensuring that the core fault handling process is not affected by communication congestion, and further guaranteeing the stable operation of the device under high load scenarios.

[0046] In some embodiments, the alarm module is deployed on the management node and each storage node, and is divided into local alarm units and centralized alarm units. The local alarm unit is deployed on each storage node and is directly connected to the detection module. It is used to handle emergency faults. When the detection module detects a sudden, high-risk anomaly on a node (such as sudden disk offline, severe mismatch of data checksum hash value, or sudden increase in hardware temperature), the local alarm unit will immediately trigger a local audible and visual alarm, such as high-frequency flashing of node indicator lights and continuous beeping of a buzzer, quickly alerting on-site maintenance personnel to the faulty node. The centralized alarm unit is deployed on the management node and directly connected to the analysis module. It is used to perform tiered push notifications based on fault levels, ensuring that maintenance personnel accurately obtain fault information. The specific tiered logic is as follows: For minor faults, only fault information is recorded and pushed to the system log, requiring no immediate intervention from maintenance personnel and reducing unnecessary maintenance interference; for moderate faults, alarms are pushed to maintenance terminals via SMS, email, or other electronic messages to remind maintenance personnel to pay attention to the repair progress and ensure that the fault is resolved within a controllable range; for severe faults, electronic messages are pushed and voice communication alarms are initiated to maintenance personnel. For example, an alarm SMS is sent first, and then the maintenance personnel are automatically dialed to report the fault details, forcibly reminding maintenance personnel to handle the emergency and ensuring that high-risk faults receive an immediate response.

[0047] In some embodiments, when the repair module completes the fault repair and generates a repair acceptance report, and the analysis module confirms that the fault has been resolved, the centralized alarm unit will automatically send an alarm resolution notification to the operation and maintenance personnel. The notification includes information such as the fault node, fault type, repair time, and repair result, so that the operation and maintenance personnel can clearly understand the complete situation of the fault handling and form an alarm closed-loop management process.

[0048] Figure 4 This is an execution flowchart of a distributed storage node fault detection and repair device provided in an embodiment of this disclosure. Figure 4As shown, this timed trigger detection is initiated by the management node, and the trigger period supports flexible configuration. The detection module then executes, where the detection task is carried out in parallel in two parts: the storage node performs local hardware, storage, and network status detection, while the management node performs global node collaborative status detection. After both types of detection are completed, the results are integrated to generate a node status report, which is then transmitted to the analysis module deployed on the management node via UDP protocol through the communication module. Upon receiving the node status report, the analysis module compares the report data with preset thresholds and a fault rule base. If the detection result is normal, the process directly enters the data archiving stage, waiting for the next detection trigger; if the detection result is abnormal, the analysis module generates a fault analysis report and marks the fault level, while simultaneously triggering the alarm module to push corresponding alarm information according to the fault level. Afterwards, processing is triaged according to the fault level: for minor faults, the storage node's repair module (i.e., the local repair unit) executes local repair strategies; for medium and severe faults, the management node's repair module (i.e., the global repair unit) coordinates global repair operations. Progress is reported in real time during the repair process, and a repair acceptance report is generated upon completion. If the repair is successful, the analysis module will confirm that the fault has been resolved, store the relevant data in the storage module, and then the process will enter the data archiving stage, waiting for the next test; if the repair fails, the alarm module of the management node will escalate the alarm to notify manual intervention. After the manual repair is completed, the process will also enter the data archiving stage.

[0049] The above process will be illustrated with a specific scenario below.

[0050] Suppose that disk 2 on storage node 3 detects 8 bad sectors. The detection module on storage node 3 generates a node status report, marking "8 bad sectors on disk 2", and sends it to the management node via the communication module. The analysis module on the management node compares the report to a threshold (8 bad sectors > 5 bad sectors), determines it as a moderate fault, generates a fault analysis report, and marks "data copy of disk 2 needs to be rebuilt". The alarm module on the management node pushes an SMS to the maintenance personnel, stating "8 bad sectors on disk 2 of storage node 3, automatic repair has been triggered". The repair module on the management node issues an instruction to copy data from storage node 5 (including the same data copy) to the spare disk of storage node 3. After the repair is completed, the repair module generates a repair acceptance report and sends it back to the analysis module. The analysis module confirms that the fault has been resolved, the data is stored in the storage module, and the alarm module pushes an "alarm clearance notification".

[0051] Based on the same inventive concept as the aforementioned distributed storage node fault detection and repair device, this application also provides a method for distributed storage node fault detection and repair in some embodiments, which utilizes the distributed storage node fault detection and repair device disclosed in the above embodiments. Figure 5This is a flowchart of a method for fault detection and repair of a distributed storage node provided in an embodiment of this disclosure. Combined with... Figure 5 The method includes the following steps: S501: Periodically triggers status detection of each storage node in the distributed storage system, collecting node status information including hardware, storage, and network status; S502: Compare the node status information with a preset fault threshold, and determine the fault level based on the rules preset in the fault rule base. The fault level includes minor fault, moderate fault and severe fault. S503: Based on the fault level and corresponding node status information, perform fault location and generate a fault analysis report containing fault node identifier, specific faulty component and scope of impact; S504: Perform the corresponding repair operation according to the fault level in the fault analysis report, and provide real-time feedback on the repair progress during the repair process. If it is a minor fault, a local repair strategy is executed at the fault node. If it is a moderate or severe fault, multiple nodes are coordinated to execute a global repair strategy. S505: If the repair is successful, a repair completion confirmation message will be generated and an alarm cancellation notification will be sent. If the repair fails, the alarm will be escalated to notify manual intervention. S506: After completing the repair result processing, return to the timed trigger detection to form a closed loop of fault detection and repair.

[0052] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this disclosure. The aforementioned storage medium can be a non-transitory storage medium, including: a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and other media capable of storing program code; it can also be a transient storage medium.

[0053] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terminology used in this application is for descriptive purposes only and is not intended to limit the scope of protection. As used in the description herein, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used herein means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0054] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0055] The methods and products (including but not limited to devices and equipment) disclosed in the embodiments herein can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection shown or discussed between each other may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to implement this embodiment according to actual needs. In addition, the functional units in the embodiments of this disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

Claims

1. An apparatus for distributed storage node failure detection and repair, the apparatus comprising: The device comprises: a detection module arranged in a management node and each storage node in a distributed storage system, for collecting node state information at a regular time and reporting the node state information to an analysis module; the analysis module is arranged in the management node, for receiving and analyzing the node state information, identifying a fault and generating a fault analysis report containing a fault level; a repair module arranged in the management node and each storage node, for executing a local repair strategy or a global repair strategy corresponding to the fault level according to the fault analysis report; a storage module arranged in the management node and each storage node, for storing system configuration, running data and user service data in an encrypted manner; an alarm module arranged in the management node and each storage node, connected with the analysis module, for triggering a hierarchical alarm according to the fault level of the fault analysis report; a communication module arranged in the management node, each storage node and a client node, connected with the detection module, analysis module, repair module, storage module and alarm module, for data transmission and instruction issuing between modules and nodes.

2. The apparatus for distributed storage node failure detection and repair of claim 1, wherein, The detection module comprises: a local detection unit arranged in each storage node, for collecting node state information of the corresponding storage node at a first configurable period, the node state information comprising: hardware state indicators, storage state indicators and network state indicators; a global detection unit arranged in the management node, for collecting and analyzing node state information of all storage nodes at a second configurable period, for inter-node collaborative state detection.

3. The apparatus for distributed storage node failure detection and repair of claim 2, wherein, The hardware state indicators comprise CPU utilization, memory occupancy, disk IO rate and disk bad track number; the storage state indicators comprise data replica integrity based on check hash value determination and storage capacity utilization; the network state indicators comprise inter-node communication delay and data packet loss rate.

4. The apparatus for distributed storage node failure detection and repair of claim 1, wherein, In the process of generating the fault analysis report, the analysis module is configured to: compare the received node state information with a preset fault threshold; determine a fault level according to rules preset in a fault rule library, wherein the fault level comprises mild fault, moderate fault and severe fault; perform fault positioning according to the fault level and corresponding node state information, to generate a fault analysis report, wherein the fault analysis report comprises fault node identification, specific fault components and influence range, for the repair module to determine a corresponding repair strategy; check historical fault data in batches every preset time, and improve the fault rule library according to the check result.

5. The apparatus for distributed storage node failure detection and repair of claim 4, wherein, The repair module comprises: a local repair unit arranged in each storage node, for executing a local repair strategy corresponding to the mild fault in the fault analysis report, wherein the local repair strategy comprises shutting down unnecessary processes, adjusting data read-write priority, deleting redundant logs or migrating cold data to a low-load node; A global repair unit is arranged in the management node and is configured to execute a global repair strategy corresponding to the moderate fault or severe fault in the fault analysis report, wherein the global repair strategy comprises copying data from a healthy node to reconstruct a damaged copy, migrating a storage task of an offline node to a backup node, or triggering a hardware replacement alarm and reconstructing data from other copies.

6. The apparatus for distributed storage node failure detection and repair of claim 5, wherein, The repair module is internally provided with a repair strategy library, which is associated with the local repair unit and the global repair unit, and is configured to automatically match a local repair strategy or a global repair strategy corresponding to the fault level according to the fault analysis report, so as to be executed by the local repair unit or the global repair unit. The repair module is further configured to feed back the progress in real time during the repair process and generate a repair acceptance report after the repair is completed and return the report to the analysis module.

7. The apparatus for distributed storage node failure detection and repair of claim 1, wherein, The storage module comprises: A node storage unit is arranged in each storage node and is configured to store user service data, local detection historical data and data copy information by using a distributed hash table. A management storage unit is arranged in the management node and is configured to store system configuration data, global fault logs and node topology information. The storage module uses AES-256 to encrypt and store data, and transmits data copies by using the SSL protocol.

8. The apparatus for distributed storage node failure detection and repair of claim 1, wherein, The communication module is configured to: For the node state information collected by the detection module, the UDP protocol is used for transmission to reduce the delay. For the repair instructions issued by the analysis module and the repair acceptance report fed back by the repair module, the TCP protocol is used for transmission to ensure reliability.

9. The apparatus for distributed storage node failure detection and repair of claim 1, wherein, The alarm module comprises: A local alarm unit is arranged in each storage node and is connected with the detection module, and is configured to immediately trigger a local audible and visual alarm when the detection module detects an emergency fault. A centralized alarm unit is arranged in the management node and is connected with the analysis module, and is configured to perform hierarchical pushing according to the fault level determined by the analysis module, including: only recording and pushing system logs for a mild fault, pushing an electronic message alarm to an operation and maintenance terminal for a moderate fault, and pushing an electronic message and starting a voice communication alarm to an operation and maintenance personnel for a severe fault. After the fault is repaired, the centralized alarm unit automatically sends an alarm release notification to form an alarm closed loop.

10. A method for distributed storage node failure detection and repair, the method comprising: The method comprises: Timing triggers the state detection of each storage node in the distributed storage system, and collects node state information including hardware, storage and network states. The node state information is compared with a preset fault threshold, and a fault level is determined based on the rules pre-stored in the fault rule library, wherein the fault level comprises a mild fault, a moderate fault and a severe fault. Based on the fault level and the corresponding node state information, fault positioning is performed to generate a fault analysis report containing fault node identification, specific fault components and influence range. According to the fault level in the fault analysis report, a corresponding repair operation is performed, and a real-time feedback of repair progress is provided during the repair process. If it is a light fault, a local repair strategy is performed at the fault node. If it is a moderate fault or a severe fault, a global repair strategy is performed by coordinating multiple nodes. If the repair is successful, repair completion confirmation information is generated and an alarm release notification is sent. If the repair fails, the alarm is upgraded to notify manual intervention. After the repair result processing is completed, a timing trigger detection is returned to form a fault detection and repair closed loop.

Images