A cloud service-based distributed storage fault diagnosis method

Abstract

The application relates to the technical field of distributed storage, in particular to a distributed storage fault diagnosis method based on cloud services, which comprises the following steps: S1, a cloud platform receives and stores a plurality of diagnosis scripts; S2, a user interface receives a search condition, and matches an associated diagnosis script based on the search condition; S3, the diagnosis script is sent to a target cluster storage node through an encrypted channel; S4, the diagnosis script is executed in a sandbox environment, a structured summary result is generated and returned to the cloud platform; S5, based on a predefined decision tree structure, the cloud platform executes a first detection script group from a root node according to the summary result, dynamically selects a second detection script group corresponding to a next node based on the first detection script group output result, and stops until reaching a leaf node; and S6, diagnosis results of all the leaf nodes are combined to generate a fault positioning report and returned to the user interface. The application enhances the fault diagnosis capability and efficiency in the distributed storage cluster.

Description

Technical Field

[0001] This invention relates to the field of distributed storage technology, and in particular to a method for diagnosing distributed storage faults and a storage medium based on cloud services. Background Technology

[0002] With the development of big data and cloud computing technologies, distributed storage has been widely used in enterprise-level storage, data backup, and big data analytics. Distributed storage achieves high availability, fault tolerance, and scalability by distributing data across multiple physical nodes. However, the complexity of the system also brings many challenges, especially in fault diagnosis and performance optimization.

[0003] Traditional fault diagnosis typically relies on manual analysis of system logs, monitoring metrics, and service call chains. This method is not only resource-intensive but also suffers from high repetition and low efficiency. Especially when the system experiences performance bottlenecks, node failures, or data loss, manual troubleshooting often struggles to pinpoint the source of the fault in a timely manner, impacting the system's stable operation.

[0004] To improve the efficiency and accuracy of fault diagnosis, automated diagnostic systems have emerged. These systems, through preset inspection items and intelligent analysis tools, can automatically collect and analyze system operating data, quickly identifying potential problems. Users only need to run the corresponding software tools to complete the initial fault investigation, significantly reducing the need for manual intervention. Furthermore, automated diagnostic systems can provide visualized analysis results, helping users to understand the system status more intuitively and improving operational efficiency.

[0005] In summary, as the scale of distributed storage systems continues to expand, traditional manual fault diagnosis methods are no longer sufficient to meet the demands of efficient operation and maintenance. Introducing automated diagnostic systems can not only reduce labor costs but also improve the speed and accuracy of fault location, ensuring the stable operation of the system. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a distributed storage fault diagnosis method based on cloud services, comprising the following steps:

[0007] S1: The cloud platform receives and stores several diagnostic scripts, including metadata information, execution logic, report templates, resource declarations, and permission requests;

[0008] S2: The user interface receives user input including the target storage node cluster identifier, fault problem classification or keyword search conditions, and matches the associated diagnostic script based on the search conditions;

[0009] S3: The diagnostic script is sent to the target cluster storage node via an encrypted channel;

[0010] S4: The target cluster storage node executes the diagnostic script in the sandbox environment, generates a structured summary result locally, and sends it back to the cloud platform;

[0011] S5: Based on a predefined decision tree structure, the cloud platform executes the first detection script group starting from the root node according to the summary results, and dynamically selects the second detection script group corresponding to the next node based on the output results of the first detection script group, until the leaf node is reached. The decision tree structure supports dynamic updates including addition, deletion and upgrading.

[0012] S6: Merge the diagnostic results of all leaf nodes to generate a fault location report and return it to the user interface.

[0013] Preferably, the metadata information, the execution logic, the report template, and the resource declaration and permission request further include:

[0014] The metadata information includes data such as script ID, name, description, applicable system version, tags, author, and category;

[0015] The execution logic includes multiple inspection methods, such as RPC calls, log analysis, metric queries, and custom tests.

[0016] The report template includes a structure for defining the diagnostic report and content including conclusions, recommendations, and a detailed analysis process;

[0017] The resource declaration and permission request are system interfaces or sensitive file paths that the diagnostic script needs to access, which must be explicitly declared.

[0018] Preferably, in step S2, the diagnostic script associated with the search criteria further includes:

[0019] S21: Based on the keyword search criteria, search the metadata information for the diagnostic script containing the same script ID, the same log keyword, or the same component name;

[0020] S22: Filter the set of diagnostic scripts under the corresponding category tags according to the fault problems, including performance bottlenecks, I / O anomalies, disk failures and network interruptions.

[0021] Preferably, in step S3, the step of sending the diagnostic script to the target cluster storage node via an encrypted channel further includes:

[0022] S31: The cloud platform uses a private key to digitally sign the diagnostic script, generating signature data containing the script hash value and timestamp;

[0023] S32: Locate the storage node topology based on the target cluster identifier, and send the diagnostic script carrying the signature data and without state dependency to the target storage node related to the fault through a two-way TLS encrypted channel;

[0024] S33: The target storage node uses a pre-set cloud public key to verify the signature data. If the hash matches and the timestamp is within the validity period, the diagnostic script is loaded; otherwise, execution is refused.

[0025] Preferably, in step S4, the target cluster storage node executes the diagnostic script in a sandbox environment, generates a structured summary result locally, and sends it back to the cloud platform, further including:

[0026] S41: Perform a pre-check on the legality, completeness, and execution permissions of the diagnostic script; if the check passes, run the diagnostic script.

[0027] S42: When running the diagnostic script in the sandbox, extract summary information including log output, standard output, performance indicators, script ID, and system call return value; calculate key indicators including performance deviation based on the summary information; and generate the structured summary result based on the key indicators.

[0028] S43: Securely transmit the structured digest results back to the cloud platform through the encrypted channel, while the original logs and performance data remain on the local node.

[0029] Preferably, in step S41, the legality, completeness, and execution permissions of the diagnostic script are pre-checked, and the diagnostic script is run after the check passes. This further includes:

[0030] Check whether the script syntax of the diagnostic script conforms to the platform script standard to perform syntax verification. If it does not conform, directly reject the transmission and return the script ID.

[0031] According to the resource declaration, the diagnostic script is verified to only access the declared resource paths in order to verify the interface whitelist. If it does not meet the requirements, the diagnostic script is isolated and a security alarm is triggered.

[0032] The diagnostic scripts containing dangerous keywords or system calls are detected and blocked to scan the blacklist. If a blacklist is found, the script is permanently added to the script library and its distribution across all platforms is blocked.

[0033] Preferably, in step S5, based on a predefined decision tree structure, the cloud platform executes the first detection script group starting from the root node according to the summary result, and dynamically selects the second detection script group corresponding to the next node based on the output result of the first detection script group, until the leaf node is reached, further including:

[0034] Based on a predefined decision tree structure, the first detection script group is executed starting from the root node according to the summary results;

[0035] The node attribute marker is detected. If the node attribute marker is a leaf node attribute, the iteration is terminated, and the diagnostic conclusions of all script groups on the path are extracted to form a local fault analysis report. If it is a non-leaf node attribute, the next node is dynamically selected based on the comparison result of the output result and the dynamic threshold. If a single iteration exceeds the set time, the process is interrupted.

[0036] Preferably, the dynamic updating of the decision tree includes:

[0037] If the dynamic update of the decision tree is to add a script, then the nodes in the decision tree are traversed. If the different states of the execution result of the added script do not affect the branch decision of the node, it is marked as an "unaffected node"; otherwise, it is marked as an "affected node". After the added script is added to the common ancestor node of the affected nodes, all affected nodes are updated. Optional update operations include adding the restriction of the execution result of the added script to the original lower-level branch conditions, or combining the execution result of the added script with the original decision conditions to add a lower-level branch. If the dynamic update of the decision tree is to delete a script, then the branches in the decision tree are traversed, all branches affected by the execution result of the deleted script are marked, the deleted script is removed from its node, and all decision conditions related to the deleted script on all branches are removed. If the decision conditions of multiple branches are consistent after removal, the two branches are merged into one branch, and the test scripts of the subordinate nodes are merged into the new node after taking the union.

[0038] If the dynamic update of the decision tree is an upgrade script, and the update does not affect the result, then it is only necessary to replace the old version with the upgrade script. If it affects the result and its judgment conditions, then it is necessary to traverse the branches in the decision tree and update all affected judgment conditions to the new values.

[0039] Preferably, the dynamic threshold further includes:

[0040] The pre-set script calculates environmental parameters, including the network card bandwidth limit and the disk IOPS limit;

[0041] Store the environmental parameters as globally shared variables;

[0042] The diagnostic script uses the globally shared variable as a judgment condition to dynamically select the next node. That is, if the network latency is greater than the network dynamic threshold, it jumps to the network diagnostic branch. If the disk error count exceeds the limit, it jumps to the hardware detection branch. If there is a duplicate detection request for the same node, the cached result is returned directly.

[0043] Based on the same concept, the present invention also provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the cloud service-based distributed storage fault diagnosis method as described in the embodiments.

[0044] Compared with the prior art, the beneficial effects of the present invention are:

[0045] This invention receives and stores several diagnostic scripts through a cloud platform, enabling online maintenance and unified release of diagnostic rules. It receives user input, including target storage node cluster identifiers, fault categories, or keyword search conditions, through a user interface. Based on the search conditions, it matches the associated diagnostic scripts, allowing users to trigger and execute diagnostic items simply by describing or selecting the problem encountered. This approach lowers the operational threshold, improves diagnostic accuracy and automation, and is key to achieving "de-expertization" problem localization compared to traditional methods requiring highly specialized personnel.

[0046] This invention loads a predefined decision tree structure and, based on the target storage node cluster and the problem type, sends a diagnostic script associated with the root node to the target cluster. This enables the script to have good maintainability and compatibility, ensuring that it will not damage the system stability during runtime. It also features a clear structure, clear semantics, and traceability, which is a key foundation for ensuring the "controllable openness" of the diagnostic system.

[0047] This invention executes diagnostic scripts in a sandbox environment at each node, generates structured summary results locally, and sends them back to the cloud platform. This achieves enhanced security and stability by using restricted permissions and an isolated environment for script execution, thus ensuring flexibility while achieving a technical balance between operational security and system reliability.

[0048] This invention is based on a predefined decision tree structure. The cloud platform executes the first detection script group starting from the root node according to the summary results. Based on the output results of the first detection script group, it dynamically selects the second detection script group corresponding to the next node, until the leaf node is reached. This allows users without any professional background to start from a simple entry point and gradually perform automated fault diagnosis. The direction of subsequent diagnosis is determined based on the results of the previous round of diagnosis, so as to gradually narrow down the problem to a very small range with as few diagnosis tasks as possible, thus accelerating the discovery of fault points.

[0049] This invention generates a fault location report by merging the diagnostic results of all leaf nodes and returns it to the user interface. This provides guidance for technical support personnel, has good user readability, and allows customers to handle some problems independently, thereby reducing after-sales intervention costs and saving software vendors a lot of human resources. Attached Figure Description

[0050] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention.

[0051] Figure 1 This is a flowchart of the distributed storage fault diagnosis method based on cloud services according to the present invention;

[0052] Figure 2 This is a flowchart of the execution script for the cloud service-based distributed storage fault diagnosis method of the present invention.

[0053] Figure 3 This is an architecture diagram of the existing storage fault diagnosis system of the present invention;

[0054] Figure 4 This is an architecture diagram of the cloud-based distributed storage fault diagnosis system of the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. Obviously, the described embodiments are only some, not all, of the embodiments described in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without creative effort are within the scope of protection of this application.

[0056] Those skilled in the art will understand that, unless otherwise stated, the singular forms “a” and “an” used herein, and “the”, may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0057] First Embodiment

[0058] Please see Figure 1 As shown, to address the issues of weak fault diagnosis capabilities, difficulty in updating tools, and low efficiency due to generalized data collection in traditional distributed storage clusters, this embodiment provides a cloud-based distributed storage fault diagnosis method. This method deploys the execution environment on each storage node, while the diagnostic logic (scripts) is centrally managed and dynamically distributed as needed, thereby achieving flexible expansion and real-time updates of diagnostic capabilities. The method includes the following steps:

[0059] S1: The cloud platform receives and stores several diagnostic scripts, including metadata information, execution logic, report templates, resource declarations, and permission requests.

[0060] Preferably, the metadata information, the execution logic, and the report template further include:

[0061] Metadata information includes script ID, name, description, applicable system version, tags, author, and category;

[0062] The execution logic includes various checking methods such as RPC calls, log analysis, metric queries, and custom tests;

[0063] The report template includes a structure for defining the diagnostic report and content including conclusions, recommendations, and a detailed analysis process;

[0064] Resource declarations and permission requests are required for scripts to explicitly declare the system interfaces or sensitive file paths they need to access (e.g., accessing / var / log / messages, calling a certain RPC). Specifically, in this embodiment, resource declarations and permission requests facilitate permission isolation and auditing. The cloud platform supports uploading and hosting detection scripts (i.e., diagnostic scripts). Each script must conform to standard interface specifications. Scripts can be jointly maintained by the cloud platform official team, enterprise developers, or community users, and version control and release auditing mechanisms are supported. The cloud platform undertakes the responsibilities of centralized management, release, verification, and lifecycle control of detection scripts.

[0065] This embodiment proposes a technical architecture that organizes fault detection logic in the form of "scripts." By centrally managing the core logic of the fault detection scripts on a cloud platform, and having the cloud platform uniformly manage, schedule, and distribute them, online maintenance and unified release of diagnostic rules are achieved. This design is significantly superior to existing technologies that embed diagnostic logic into cluster nodes for fixed deployment, or rely on fixed code or embedded probes for operation and maintenance diagnosis, in terms of diagnostic logic updates, deployment flexibility, scalability, and maintainability. Because the diagnostic scripts are uniformly managed in the cloud, the platform supports regular updates, version control, and multi-version compatibility matching, enabling the logic for handling new problems to quickly cover all cluster nodes without manual intervention or system restarts. This cloud-edge collaborative architecture greatly improves system response speed and maintenance efficiency. By decoupling the diagnostic logic from the platform, the platform can continuously evolve its diagnostic capabilities without upgrading system components.

[0066] Secondly, this embodiment supports joint participation in script development by developers and community users, and manages the script through metadata information (such as applicable version, problem background, dependent components, etc.), enabling continuous expansion and refinement of the problem database. This allows the system to adapt to complex and ever-changing distributed storage scenarios and quickly respond to new diagnostic needs. In contrast, the predefined diagnostic logic in traditional solutions often struggles to cover sudden problems or operational differences brought about by cross-version evolution, making this embodiment significantly superior to traditional solutions in terms of diagnostic coverage and adaptability.

[0067] S2: The user interface receives user input including the target storage node cluster identifier, fault problem category, or keyword search conditions. Based on the search conditions, it matches associated diagnostic scripts. Specifically, in this embodiment, the user interface provides a graphical operation entry point, including: Problem category navigation: such as performance problems, I / O anomalies, disk failures, network bottlenecks, etc., mainly to help users find associated diagnostic scripts to try running and locate the problem when they have a general judgment on the problem; Search function: supports quick search for related scripts by keywords (such as error code, error log, component name). Each script has associated script ID, error log, component name, and other information, and this information will be updated in real time in the cloud.

[0068] When users see a script ID or error log, they can directly find the relevant diagnostic script through this information; Script execution entry point: Allows users to manually trigger diagnostic tasks. After the user selects the script and target cluster, the diagnostic script will be sent to the nodes of the specified cluster for execution.

[0069] Preferably, in step S2, the diagnostic script based on the search criteria and associated results further includes:

[0070] S21: Based on the keyword search conditions, search the metadata information for diagnostic scripts containing the same script ID, the same log keyword, or the same component name. Specifically, in this embodiment, a metadata inverted index is established, and the index key includes: script ID field (primary key index), log keyword field (full-text index), and component name field (prefix index). The keyword K input by the user is received, a multi-field joint search is performed, and a set of matching scripts is output.

[0071] S22: Filter the collection of diagnostic scripts under the corresponding category tags based on the fault problems, including performance bottlenecks, I / O anomalies, disk failures, and network interruptions.

[0072] Receive keyword K input by the user and perform a multi-field joint search, including fuzzy matching and prefix matching. The fuzzy matching is as follows:

[0073] A word vector space model is constructed for the log keyword field, and the similarity between keyword K and log keyword L is calculated using the following formula: as follows:

[0074]

[0075] in, Indicator, TF-IDF represents a word that belongs to both keyword K and log keyword L. TF-IDF weight values, This represents the sum of the TF-IDF values ​​of the intersection word w. It is the product of the Euclidean norms of the TF-IDF vectors of K and L (i.e., the normalization factor).

[0076] A match is determined when sim(K,L) > threshold θ;

[0077] It supports wildcard queries. *K matches keywords that end with K, K* matches keywords that start with K, and *K* matches keywords that contain K.

[0078] This embodiment is guided by typical problem scenarios (such as performance issues, I / O anomalies, disk failures, etc.), constructing a thematic script knowledge base, and automatically recommending corresponding detection items through a user interface (UI) or keyword search. Users only need to describe or select the problem they encounter to trigger and execute diagnostic items. This approach lowers the operational threshold and improves the accuracy and automation of diagnosis. Compared with traditional methods that require highly specialized personnel, this technology is key to achieving "de-expertized" problem localization.

[0079] S3: The diagnostic script is sent to the target cluster storage node through an encrypted channel. Specifically, in this embodiment, after the user selects the target node and the corresponding diagnostic script through the platform interface or API, the selected script will be sent to the local runtime module of the specified node through a secure control channel (i.e., an encrypted channel). The control channel can be established based on an encrypted communication protocol (such as TLS / SSL) to ensure the security and integrity of the diagnostic script transmission and prevent the data from being tampered with or eavesdropped on during transmission.

[0080] Preferably, in step S3, the diagnostic script is sent to the target cluster storage node via an encrypted channel, further including:

[0081] S31: The cloud platform uses a private key to digitally sign the diagnostic script, generating signature data containing the script hash value and timestamp;

[0082] S32: Locate the storage node topology based on the target cluster identifier, and send the diagnostic script carrying the signature data and without state dependency to the target storage node related to the fault through a bidirectional TLS encrypted channel. Specifically, in this embodiment, the bidirectional TLS encrypted channel establishment process includes the target storage node providing a client certificate containing a cluster identifier, the cloud platform verifying the consistency between the cluster identifier of the client certificate and the target cluster identifier, the channel encryption algorithm being AES-256-GCM, if it is a disk failure, then all storage nodes equipped with SSDs are located, if it is a network interruption, then the core switch directly connected nodes are located, and a script packet with an additional topology routing identifier is sent, which is then forwarded to the final node by the cluster gateway.

[0083] S33: The target storage node uses a preset cloud public key to verify the signature data. If the hash matches and the timestamp is within the validity period, the diagnostic script is loaded; otherwise, execution is refused. Specifically, in this embodiment, if the signature hash does not match, an alarm code indicating script tampering is sent to the cloud platform. If the timestamp expires, an alarm code indicating that the preset validity period has been exceeded is sent to the cloud platform. If the cloud platform receives the code indicating script tampering, it adds the diagnostic script to the malicious script library and blocks its distribution across the entire platform.

[0084] S4: The target cluster storage node executes the diagnostic script in the sandbox environment, generates a structured summary result locally, and sends it back to the cloud platform. Specifically, in this embodiment, the target cluster storage node deploys a lightweight, sandboxed execution environment with multiple functions such as script loading, running, resource isolation, and result feedback. After receiving the script, the target cluster storage node will pre-check the script's legality, completeness, and running permissions, and start the execution process after passing the verification. During execution, the runtime module can restrict the scope of the script's access to system resources (such as network, files, memory, etc.) to avoid affecting normal business operations. Secondly, the platform supports version control and parallel execution of multiple copies of the distributed script, which can realize rapid batch scheduling of the same script on multiple nodes. The script distribution process not only has high reliability and security, but also achieves fine-grained control and traceability of the diagnostic process, significantly improving the availability and maintainability of the system's automated diagnosis.

[0085] Preferably, in step S4, the target cluster storage node executes a diagnostic script in a sandbox environment, generates a structured summary result locally, and sends it back to the cloud platform, further including:

[0086] S41: Perform a pre-check on the legality, completeness, and execution permissions of the diagnostic script. Run the diagnostic script after the check passes.

[0087] S42: When running diagnostic scripts in the sandbox, extract summary information including log output, standard output, performance metrics, script ID, and system call return values. Calculate key metrics, including performance deviation, based on the summary information. Generate structured summary results based on the key metrics. Specifically, in this embodiment, RPC calls are used (calling local RPC interfaces (such as monitoring interfaces, disk status) to query node status information, parse system log files (to determine whether they are structured or unstructured) to retrieve abnormal events, collect Prometheus data, and collect performance metric data. After each script upload or update, the platform automatically adds it to the continuous integration testing process, automatically performing regression testing in multiple test environments, including simulated nodes in storage clusters with different versions and load states. Custom tests are executed, including simulating I / O read / write operations to measure latency, injecting test packets to detect network connectivity, performing stress tests to verify resource bottlenecks, and obtaining running results. The compatibility information is verified to ensure it matches the current node version, and the expected output is confirmed. To ensure consistent and stable script behavior, all platform-hosted scripts are executed regularly (e.g., daily / weekly) in a dedicated test cluster to verify their continued functionality after system updates. Scripts marked as "recommended" or "high-frequency" will be executed even more frequently. All scripts are version-managed, with each release recording changes, release time, and reviewer. Users can freely choose to execute the latest version or specify historical versions to reproduce issues in specific environments. The platform provides a "risk assessment report" for each version, informing users whether the script has passed CI verification and whether there are any failure records. Sandbox mechanisms (such as containers) are also supported to ensure that script execution does not affect the storage service's performance.

[0088] S43: Securely transmit structured digest results back to the cloud platform via an encrypted channel, while the original logs and performance data remain on the local node.

[0089] To ensure the compatibility, security, and controllability of distributed scripts, this embodiment defines a standardized script framework specification, including but not limited to: structured requirements such as execution logic entry points, version compatibility annotations, report format generation, and problem background descriptions. Scripts must declare their purpose and applicable conditions and be verified through a cloud platform or continuous deployment system. This mechanism enables scripts to have good maintainability and compatibility, ensuring that runtime will not damage system stability. It also features clear structure, semantic clarity, and traceability, forming a key foundation for ensuring the "controllable openness" of the diagnostic system.

[0090] Preferably, in step S41, the legality, completeness, and execution permissions of the diagnostic script are pre-checked. After the check passes, the diagnostic script is run, further including:

[0091] Check whether the script syntax of the diagnostic script conforms to the platform script standard to perform syntax validation. If it does not conform, directly reject the transmission and return the script ID.

[0092] The diagnostic script is verified to access only the declared resource paths according to the resource declaration, so as to implement the verification interface whitelist. If it does not meet the requirements, the diagnostic script is isolated and a security alarm is triggered.

[0093] Diagnostic scripts containing dangerous keywords or system calls are detected and blocked to scan blacklists. If a script is found to be in the blacklist, it is permanently added to the script library and its distribution across all platforms is blocked.

[0094] There is a predefined decision tree structure that can be built based on a large number of historical cases and expert experience. It aims to achieve rapid convergence and root cause identification of system faults with as few diagnostic steps as possible, and is used for efficient and progressive fault detection and problem localization of target systems.

[0095] This embodiment does not adopt the traditional "full data collection and re-analysis" framework. Instead, it uses a script-on-demand execution design to precisely focus on the time period, indicator range, and specific symptoms that users care about. This avoids the resource waste and performance interference caused by collecting large amounts of irrelevant data. Furthermore, the platform uses automated regression testing and continuous deployment mechanisms to perform static analysis and dynamic execution verification of the scripts, ensuring that all deployed scripts do not adversely affect the stability and performance of the cluster. It significantly outperforms traditional solutions in terms of data collection efficiency and resource friendliness.

[0096] To prevent the introduction of risks or misjudgments, this embodiment designs an automated script verification mechanism based on continuous deployment (CI / CD). Newly uploaded scripts must undergo an automated testing process, simulating operation in various environments and performing periodic verifications to confirm that their behavior meets expectations, resource usage is controlled, and they have no negative impact on the system. Furthermore, in the runtime module, script execution employs restricted permissions and isolated environments (such as container sandboxes and resource quotas) to enhance security and stability. This mechanism achieves a technical balance between operational security and system reliability while ensuring flexibility.

[0097] S5: Based on a predefined decision tree structure, the cloud platform executes the first detection script group starting from the root node according to the summary results. Based on the output results of the first detection script group, it dynamically selects the second detection script group corresponding to the next node, until the leaf node is reached. The decision tree structure supports dynamic updates including addition, deletion, and upgrade. Specifically, in this embodiment, each node of the decision tree corresponds to a set of detection scripts. Each set of scripts is used to detect the health status and abnormal behavior of a specific sub-module, component, or path in the system. When the user triggers a detection operation, the corresponding detection scripts will be executed sequentially starting from the root node of the decision tree. The scripts will be distributed to the relevant nodes in the cluster and run locally. At the same time, the key fields or summary information in each detection result are aggregated to the cloud diagnostic platform in real time. Based on the diagnostic results returned by the current node and combined with the decision tree logic, the cloud platform dynamically decides the detection scripts to be executed in the next stage and their distribution positions.

[0098] Existing test scripts and expert knowledge are organized into a decision tree structure and then deployed into the system. This allows users without any professional background to start from a simple entry point and gradually perform automated troubleshooting. Based on the results of the previous round of troubleshooting, the system determines the direction of subsequent troubleshooting, narrowing down the problem to a very small range with as few troubleshooting tasks as possible, thus accelerating the discovery of the fault point.

[0099] Please see Figure 2 As shown, in step S5, based on a predefined decision tree structure, the cloud platform executes the first detection script group starting from the root node according to the summary results, and dynamically selects the second detection script group corresponding to the next node based on the output results of the first detection script group, until the leaf node is reached, further including:

[0100] Based on a predefined decision tree structure, the first detection script group is executed starting from the root node according to the summary results;

[0101] The detection node attribute markers are checked. If the node attribute marker is a leaf node attribute, the iteration is terminated, and the diagnostic conclusions of all script groups for that path are extracted to form a local fault analysis report. If the attribute is not a leaf node attribute, the next node is dynamically selected based on the comparison results of the output results and the dynamic threshold. If a single iteration exceeds the set time, the process is interrupted. Specifically, in this embodiment, the detection sequence formed by each path can be regarded as a directed fault analysis trajectory. Finally, all trajectories are merged to form a structured diagnostic report. The report clearly marks the abnormal indicators, triggering conditions, path branching basis, and preliminary root cause analysis for users to manually confirm and intervene.

[0102] Preferably, the dynamic updating of the decision tree includes:

[0103] If the dynamic update of the decision tree is to add a script, then the nodes in the decision tree are traversed. If the different states of the execution result of the new script will not affect the branch decision of the node, it is marked as an "unaffected node". Otherwise, it is marked as an "affected node". After the new script is added to the common ancestor node of the affected nodes, all affected nodes are updated. Optional update operations include adding the restriction of the execution result of the new script to the original lower-level branch conditions, and combining the execution result of the new script with the original decision conditions to add a lower-level branch. If the dynamic update of the decision tree is to delete a script, then the branches in the decision tree are traversed. All branches affected by the execution result of the deleted script are marked. The deleted script is removed from its node. At the same time, all decision conditions related to the deleted script on the branches are removed. If the decision conditions of multiple branches are consistent after removal, the two branches are merged into one branch. The test scripts of the subordinate nodes are merged into the new node after taking the union.

[0104] If the dynamic update of the decision tree is an upgrade script, and the update does not affect the result, then you only need to replace the old version with the upgrade script. If it affects the result and its decision conditions, then you need to traverse the branches in the decision tree and update all the affected decision conditions to the new values.

[0105] Preferably, the dynamic threshold further includes:

[0106] The pre-set script calculates environmental parameters, including the network card bandwidth limit and the disk IOPS limit;

[0107] Store environment parameters as globally shared variables;

[0108] The diagnostic script uses globally shared variables as judgment conditions to dynamically select the next node. That is, if the network latency is greater than the network dynamic threshold, it jumps to the network diagnostic branch; if the disk error count exceeds the limit, it jumps to the hardware detection branch. If there are repeated detection requests for the same node, the cached result is returned directly. Specifically, in this embodiment, there is a shared storage area among all scripts. This storage area stores some key variables in the form of kev-value. The technical parameters and judgment conditions in the inspection process can be determined by some predefined global variables, rather than a specific value. By default, the global variables that each script needs to expose will be marked in the script. For example, in a performance problem inspection script, it is necessary to check whether the write rate of a storage node has reached the limit. However, since the write rate limit varies in different environments and needs to be calculated in a complex way, the platform will pre-set a script to detect information such as the network card bandwidth, disk model, and configuration parameters of the storage cluster. Then, it will obtain the estimated values ​​of the performance limit of each layer within the node through the cloud platform's database and store them in the shared variables. In the aforementioned performance problem inspection script, the shared variables can be directly referenced as the basis for determining whether the bandwidth has reached the limit.

[0109] By employing a step-by-step reasoning and interactive script distribution mechanism, the scope of system fault investigation is efficiently narrowed down, thereby quickly locating the root cause of potential problems with minimal resource consumption, significantly improving the automation level of system maintenance and the accuracy of diagnosis.

[0110] S6: Merge the diagnostic results of all leaf nodes to generate a fault location report and return it to the user interface. Specifically, in this embodiment, the analysis conclusion is based on the diagnostic results (such as latency p99>10ms, error count>threshold) (whether there is an anomaly, and if so, the severity is determined). Based on the analysis results, repair suggestions are proposed (restarting nodes, upgrading software, adjusting parameters, replacing disks, etc.). The report (including the problem background, detection process, data analysis results, and suggested measures) is sent back to the management platform within the cluster for users to view. It will not be disseminated to the public network, ensuring data security.

[0111] This embodiment provides a visualized inspection item list, automatic distribution, and report generation functions, enabling users, even those without knowledge of the system's internal implementation details, to troubleshoot and initially handle complex problems through simple operations. Compared to existing technologies, the fault location report in this embodiment not only provides guidance for technical support personnel but also boasts good user readability, facilitating customers to handle some issues independently, thereby reducing after-sales intervention costs and effectively lowering the frequency of manual intervention. It avoids a large number of repetitive operations, giving the system a preliminary capability of "self-diagnosis and self-repair," significantly outperforming traditional solutions in terms of user experience and operational cost control.

[0112] The following is a typical usage process of this embodiment:

[0113] During a cluster maintenance, service support personnel noticed that some storage nodes experienced a decline in business performance within a specific time period. Further analysis revealed that this performance degradation was usually accompanied by two characteristics: first, abnormally high read and write bandwidth loads on the nodes; and second, warning messages indicating that the rate limiting mechanism had been triggered could be observed in the node's system logs. In this case, although the underlying hardware was not faulty, the high pressure on resource allocation affected other normal services, resulting in a decline in overall business quality.

[0114] To address this type of issue and achieve rapid reproduction and automated localization, the developers wrote a dedicated detection script based on the platform's scripting framework. The script's workflow is roughly as follows: First, it calls the Metric service within the cluster via the platform's RPC interface to obtain read / write bandwidth data for each storage node within the target time range. Then, the script identifies nodes whose bandwidth metrics significantly deviate from the baseline level during this time period and initially marks them as potential anomalous nodes.

[0115] Next, the script parses the system logs of these abnormal nodes to search for alarm information related to the rate limiting mechanism. By aligning the timestamps, the script further filters out the key time periods that both have bandwidth anomalies and trigger rate limiting alarms. This cross-analysis process effectively improves the accuracy of problem localization and avoids misjudgments that may be caused by relying solely on bandwidth data.

[0116] After identifying the abnormal time period and nodes, the script will further query the IO request status of the clients that interacted with these nodes during the corresponding time period, identify the clients with significant bandwidth consumption, and record them in detail. All analysis results are finally compiled into a report, which includes abnormal nodes, abnormal time periods, log fragments that triggered rate limiting, a list of high-bandwidth clients involved, etc., and also includes operation suggestions: for example, it is recommended that users further investigate the business background of these clients that generate high load, and if necessary, implement temporary rate limiting on relevant clients through platform configuration, so as to avoid cascading impact on global business.

[0117] Once completed, the script was categorized as a "performance issue" detection tool and deployed to the cloud-based script center via the platform's publishing mechanism. When users encounter similar performance bottlenecks, they simply select the corresponding check item on the platform, which automatically distributes the script to the cluster for execution, completing the entire process of data collection, analysis, and report generation without the need for after-sales engineers. This not only significantly shortens the troubleshooting time but also greatly enhances the cluster's self-diagnostic capabilities, enabling the discovery and initial handling of performance issues.

[0118] Second Embodiment

[0119] Please see Figure 3 As shown, a typical automated diagnostic system architecture is as follows:

[0120] Fault diagnosis request access: The user or system management module initiates a fault diagnosis request for the device or cluster.

[0121] Data collection and analysis: The system extracts storage load data and interaction record data from the detection feedback information, and judges the status of each node or storage module based on these data, thereby locating the scope of the fault impact, namely the target storage module and the target cluster device.

[0122] Diagnostic result output: After the analysis steps, a fault location result is finally generated for subsequent processing or display.

[0123] Although the above solutions can achieve diagnosis and fault location in a distributed environment, they still have the following key drawbacks:

[0124] 1. Fixed deployment of diagnostic logic, inconvenient updates: In existing solutions, diagnostic logic is often deployed directly on nodes or services within the cluster. However, the main problems with this approach are: every modification or optimization of the diagnostic logic requires redeployment of the service, resulting in high update costs; it lacks flexibility and makes it difficult to respond promptly to new problems or environmental changes; once deployed to the production environment, the logic is in a "fixed" state, which is not conducive to continuous evolution and remote management.

[0125] 2. Low user participation and closed diagnostic tools: The diagnostic process is entirely automated by the platform, and users cannot obtain diagnostic tools or information for a specific fault scenario as needed; it does not support customizing, downloading, or calling tools by problem, which limits users' initiative and ability to expand their capabilities.

[0126] 3. Generalized data collection leads to high resource consumption: Existing solutions adopt a framework design of "collect data first, then analyze," which often defaults to collecting a large amount of operational data from all nodes (such as IO load, CPU, network, storage status, etc.). Before the specific fault direction is determined, extensive data collection is carried out, resulting in a large amount of invalid or low-value data being processed. This not only increases the computing and network load, but may also interfere with normal business operations, affecting diagnostic efficiency and system performance.

[0127] Please refer to Figure 4 As shown, to solve the above problems, this invention provides a distributed storage fault diagnosis system based on cloud services, comprising:

[0128] The receiving module receives and stores several diagnostic scripts on the cloud platform, including metadata information, execution logic, report templates, resource declarations, and permission requests. The metadata information includes data such as script ID, name, description, applicable system version, tags, author, and category. The execution logic includes various checking methods such as RPC calls, log analysis, metric queries, and custom tests. The report template includes content for defining the structure of the diagnostic report, including conclusions, recommendations, and detailed analysis process.

[0129] The matching module receives user input, including the target storage node cluster identifier, fault problem classification, or keyword search conditions, and matches the associated diagnostic script based on the search conditions.

[0130] The distribution module distributes the diagnostic script to the target cluster storage node via an encrypted channel;

[0131] Execution module: The target cluster storage node executes the diagnostic script in the sandbox environment, generates structured summary results locally, and sends them back to the cloud platform;

[0132] Detection module: Based on a predefined decision tree structure, the cloud platform executes the first detection script group starting from the root node according to the summary results, and dynamically selects the second detection script group corresponding to the next node based on the output results of the first detection script group, until the leaf node is reached;

[0133] Report generation module: Merges the diagnostic results of all leaf nodes to generate a fault location report and returns it to the user interface.

[0134] This embodiment achieves centralized updating, rapid deployment, and elastic scaling of detection logic through unified management and scheduling of detection scripts via a cloud platform, greatly improving the automation and intelligence of problem diagnosis. However, a feasible alternative solution that does not rely on a cloud platform can also be provided.

[0135] The core idea of ​​this alternative is to deploy a resident diagnostic agent module on each storage node. All detection logic, execution strategies, and update mechanisms are completed independently locally, without relying on the cloud platform. In actual deployment, this alternative path can be implemented in the following ways:

[0136] First, the detection scripts or modules can be delivered to the cluster nodes via offline packages by after-sales support personnel or the customer's own operations and maintenance personnel. They are then deployed to the local diagnostic agent on each node through installation or script registration. This agent supports scheduled scans for updates, version verification, and rollback mechanisms, ensuring that even in isolated environments without cloud communication capabilities or with strong security controls, it still possesses certain update capabilities and script lifecycle management capabilities.

[0137] Third Embodiment

[0138] Based on the same concept, this embodiment also provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of a cloud-based distributed storage fault diagnosis method according to an embodiment of the present invention.

[0139] It is understood that, for the aforementioned cloud-based distributed storage fault diagnosis method, if all components are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer server or a network device, etc.) to execute all or part of the steps of the methods in the various embodiments of this invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0140] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0141] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for fault diagnosis of distributed storage based on cloud services, characterized in that, Includes the following steps: S1: The cloud platform receives and stores several diagnostic scripts, including metadata information, execution logic, report templates, resource declarations, and permission requests; S2: The user interface receives user input including the target storage node cluster identifier, fault problem classification or keyword search conditions, and matches the associated diagnostic script based on the search conditions; S3: The diagnostic script is sent to the target cluster storage node via an encrypted channel; S4: The target cluster storage node executes the diagnostic script in the sandbox environment, generates a structured summary result locally, and sends it back to the cloud platform; S5: Based on a predefined decision tree structure, the cloud platform executes the first detection script group starting from the root node according to the summary results, and dynamically selects the second detection script group corresponding to the next node based on the output results of the first detection script group, until the leaf node is reached. The decision tree structure supports dynamic updates including addition, deletion and upgrading. S6: Merge the diagnostic results of all leaf nodes to generate a fault location report and return it to the user interface.

2. The distributed storage fault diagnosis method based on cloud services according to claim 1, characterized in that, In step S5, based on a predefined decision tree structure, the cloud platform executes the first detection script group starting from the root node according to the summary results, and dynamically selects the second detection script group corresponding to the next node based on the output results of the first detection script group, until the leaf node is reached, further including: Based on a predefined decision tree structure, the first detection script group is executed starting from the root node according to the summary results; The node attribute marker is detected. If the node attribute marker is a leaf node attribute, the iteration is terminated, and the diagnostic conclusions of all script groups in the path are extracted to form a local fault analysis report. If it is a non-leaf node attribute, the next node is dynamically selected based on the comparison result of the output result of the first detection script group and the dynamic threshold. If a single iteration exceeds the set time, the process is interrupted.

3. The distributed storage fault diagnosis method based on cloud services according to claim 2, characterized in that, The decision tree structure supports new dynamic updates, including: If the dynamic update of the decision tree structure is a new script, then the nodes in the decision tree are traversed. If the different states of the execution result of the new script will not affect the branch decision of the node, it is marked as an "unaffected node"; otherwise, it is marked as an "affected node". After the new script is added to the common ancestor node of the affected nodes, all affected nodes are updated. Optional update operations include adding the restriction of the execution result of the new script to the original lower-level branch conditions, and combining the execution result of the new script with the original decision conditions to add a lower-level branch.

4. The distributed storage fault diagnosis method based on cloud services according to claim 2, characterized in that, The decision tree structure supports dynamic updates for deletions, including: If the dynamic update of the decision tree structure is a deletion script, then the branches in the decision tree are traversed, all branches affected by the execution result of the deletion script are marked, the deletion script is removed from its node, and all decision conditions related to the deletion script on all branches are removed. If the decision conditions of multiple branches are consistent after removal, the two branches are merged into one branch, and the test scripts of the subordinate nodes are merged into the new node after taking the union.

5. The distributed storage fault diagnosis method based on cloud services according to claim 2, characterized in that, The decision tree structure supports dynamic updates for upgrades, including: If the dynamic update of the decision tree structure is an upgrade script, and the update does not affect the result, then it is only necessary to replace the old version with the upgrade script. If it affects the result and its judgment conditions, then it is necessary to traverse the branches in the decision tree and update all affected judgment conditions to the new values.

6. The distributed storage fault diagnosis method based on cloud services according to claim 1, characterized in that, In step S3, the step of sending the diagnostic script to the target cluster storage node via an encrypted channel further includes: S31: The cloud platform uses a private key to digitally sign the diagnostic script, generating signature data containing the script hash value and timestamp; S32: Locate the storage node topology based on the target cluster identifier, and send the diagnostic script carrying the signature data and without state dependency to the target storage node related to the fault through a two-way TLS encrypted channel; S33: The target storage node uses a pre-set cloud public key to verify the signature data. If the hash matches and the timestamp is within the validity period, the diagnostic script is loaded; otherwise, execution is refused.

7. The distributed storage fault diagnosis method based on cloud services according to claim 1, characterized in that, The metadata information, the execution logic, the report template, and the resource declaration and permission request further include: The metadata information includes data such as script ID, component name, description, applicable system version, tags, author, and category. The execution logic includes multiple inspection methods, such as RPC calls, log analysis, metric queries, and custom tests. The report template includes a structure for defining the diagnostic report and content including conclusions, recommendations, and a detailed analysis process; The resource declaration and permission request are system interfaces or sensitive file paths that the diagnostic script needs to access, which must be explicitly declared.

8. The distributed storage fault diagnosis method based on cloud services according to claim 7, characterized in that, In step S2, the target cluster storage node executes the diagnostic script in a sandbox environment, generates a structured summary result locally, and sends it back to the cloud platform, further including: S41: Perform a pre-check on the legality, completeness, and execution permissions of the diagnostic script; if the check passes, run the diagnostic script. S42: When running the diagnostic script in the sandbox, extract summary information including log output, standard output, performance indicators, script ID, and system call return value; calculate key indicators including performance deviation based on the summary information; and generate the summary result based on the key indicators. S43: The digest results are securely transmitted back to the cloud platform through the encrypted channel, while the original logs and performance data are kept locally on the node.

9. The distributed storage fault diagnosis method based on cloud services according to claim 8, characterized in that, In step S41, the legality, completeness, and execution permissions of the diagnostic script are pre-checked. If the check passes, the diagnostic script is executed. This further includes: Check whether the script syntax of the diagnostic script conforms to the platform script standard to perform syntax verification. If it does not conform, directly reject the transmission and return the script ID. According to the resource declaration, the diagnostic script is verified to only access the declared resource paths in order to verify the interface whitelist. If it does not meet the requirements, the diagnostic script is isolated and a security alarm is triggered. The diagnostic scripts containing dangerous keywords or system calls are detected and blocked to scan the blacklist. If a blacklist is found, the script is permanently added to the script library and its distribution across all platforms is blocked.

10. The distributed storage fault diagnosis method based on cloud services according to claim 7, characterized in that, In step S2, the diagnostic script, which is matched and associated based on the search conditions, includes: S21: Based on the keyword search criteria, search the metadata information for the diagnostic script that contains the same script ID, the same log keyword, or the same component name; S22: Filter the set of diagnostic scripts under the corresponding category tags according to the fault problems, including performance bottlenecks, I / O anomalies, disk failures and network interruptions.

11. The distributed storage fault diagnosis method based on cloud services according to claim 2, characterized in that, The dynamic threshold further includes: The pre-set script calculates environmental parameters, including the network card bandwidth limit and the disk IOPS limit; Store the environmental parameters as globally shared variables; The diagnostic script uses the globally shared variable as a judgment condition to dynamically select the next node. That is, if the network latency is greater than the network dynamic threshold, it jumps to the network diagnostic branch. If the disk error count exceeds the limit, it jumps to the hardware detection branch. If there is a duplicate detection request for the same node, the cached result is returned directly.

12. A storage medium storing computer-readable instructions, characterized in that, When the computer-readable instructions are executed by one or more processors, the one or more processors perform the steps of the cloud service-based distributed storage fault diagnosis method as described in any one of claims 1 to 11.

Images