Fault processing method and device for storage system

Abstract

The invention relates to the field of data processing, in particular to a fault processing method for a storage system, and the method comprises the steps: periodically obtaining a snapshot file of a first cluster, and loading the snapshot file to a second cluster; the first cluster and the second cluster are storage systems of the same type; after it is detected that the first cluster breaks down, a first log file is obtained from the first cluster; the first log file is used for recording data operation on the first cluster; comparing the first log file with a snapshot file of the first cluster in the latest period to obtain a first incremental file which is increased compared with the backup file in the first log file; the first incremental file is loaded to the second cluster; switching an access point of a target service from the first cluster to the second cluster; the target service comprises a service running in the first cluster before the first cluster breaks down.

Description

Technical Field

[0001] This application relates to the field of data processing, and particularly to a method and apparatus for handling failures of a storage system. Background Art

[0002] Currently, with the development of cloud computing, many services have started to use storage systems to store and manage critical information required for service operation.

[0003] Currently, when a storage system fails, a common failure handling method is to switch services from the original storage system with a failure to a backup storage system. However, in the prior art, on the one hand, the recovery time objective (RTO) of the prior art is relatively long, and it takes a relatively long time to switch services from the original storage system to the backup storage system so that the services can resume operation; on the other hand, the recovery point objective (RPO) of the prior art is relatively long, and the backup storage system may only recover to the state of the original storage system a long time ago.

[0004] Therefore, how to shorten the durations of RTO and RPO when a storage system fails is a problem that needs to be solved currently. Summary of the Invention

[0005] To solve the above technical problems, this application provides a method and apparatus for handling failures of a storage system, which are used to quickly resume services to normal when the storage system fails.

[0006] To achieve the above object, this application provides the following technical solutions:

[0007] In a first aspect, a method for handling failures of a storage system is provided. The method includes: periodically obtaining snapshot files of a first cluster and loading the snapshot files into a second cluster; the first cluster and the second cluster are storage systems of the same type; after detecting that the first cluster fails, obtaining a first log file from the first cluster; the first log file is used to record data operations on the first cluster; comparing the first log file with the snapshot file of the first cluster in the most recent period to obtain a first incremental file that is increased in the first log file compared to the snapshot file of the first cluster in the most recent period; loading the first incremental file into the second cluster; switching the access point of a target service from the first cluster to the second cluster; the target service includes services that ran in the first cluster before the first cluster failed.

[0008] In some implementations, loading the snapshot file to the second cluster includes: comparing a first snapshot file with a second snapshot file to obtain a second incremental file that is added to the first snapshot file compared to the second snapshot file; wherein the first snapshot file is the snapshot file corresponding to the first cluster in the current cycle, and the second snapshot file is the snapshot file in the cycle before the first snapshot file; and loading the second incremental file to the second cluster.

[0009] In some implementations, after detecting a failure in the first cluster, obtaining a first log file from the first cluster includes: after detecting a failure in the first cluster, reading the log files recorded in multiple nodes in the first cluster to obtain multiple log file copies; determining the first log file according to the multiple log file copies; the first log file includes: the file parts that are recorded in all of the multiple log file copies.

[0010] In some implementations, after detecting a failure in the first cluster, obtaining a first log file from the first cluster includes: after detecting a failure in the first cluster, reading the log files recorded in multiple nodes in the first cluster to obtain multiple log file copies; determining the first log file according to the multiple log file copies; the first log file includes: the file parts that are recorded in all of the multiple log file copies, and the different file parts included in each log file copy respectively.

[0011] In some implementations, after detecting a failure in the first cluster, obtaining a first log file from the first cluster includes: after detecting a failure in the first cluster, reading the log files recorded in multiple nodes in the first cluster to obtain multiple log file copies; displaying the different file parts included in each of the multiple log file copies on an operation interface; determining a first log file according to a first operation on the operation interface; the first operation is used to indicate selecting a target different file part from the different file parts included in each log file copy; the first log file includes: the file parts that are recorded in all of the multiple log file copies, and the target different file part.

[0012] In some implementations, the first cluster and the second cluster are deployed in different container management clusters; or, the first cluster and the second cluster are deployed in different physical machines in the same container management cluster.

[0013] In some implementations, the container management cluster is a kubernetes cluster.

[0014] In some implementations, the first cluster and the second cluster are etcd clusters with the same version.

[0015] In a second aspect, a fault handling device for a storage system is provided, including: a loading unit configured to periodically obtain a snapshot file of a first cluster and load the snapshot file into a second cluster; the first cluster and the second cluster being storage systems of the same type; an obtaining unit configured to, after detecting a fault in the first cluster, obtain a first log file from the first cluster; the first log file being used to record data operations on the first cluster; a comparison unit configured to compare the first log file with the snapshot file of the first cluster in the most recent period to obtain a first incremental file that has been added to the first log file compared to the snapshot file of the first cluster in the most recent period; the loading unit further configured to load the first incremental file into the second cluster; a switching unit configured to switch an access point of a target service from the first cluster to the second cluster; the target service including services that were running in the first cluster before the first cluster failed.

[0016] In some implementations, loading the snapshot file into the second cluster includes: comparing a first snapshot file with a second snapshot file to obtain a second incremental file that has been added to the first snapshot file compared to the second snapshot file; where the first snapshot file is the snapshot file corresponding to the first cluster in the current period, and the second snapshot file is the snapshot file in the period before the first snapshot file; and loading the second incremental file into the second cluster.

[0017] In some implementations, the obtaining unit configured to, after detecting a fault in the first cluster, obtain a first log file from the first cluster includes: the obtaining unit configured to, after detecting a fault in the first cluster, read log files recorded in multiple nodes in the first cluster to obtain multiple log file copies; the obtaining unit further configured to determine the first log file based on the multiple log file copies; the first log file including: a file portion that is recorded in all of the multiple log file copies.

[0018] In some implementations, the obtaining unit configured to, after detecting a fault in the first cluster, obtain a first log file from the first cluster includes: the obtaining unit configured to, after detecting a fault in the first cluster, read log files recorded in multiple nodes in the first cluster to obtain multiple log file copies; the obtaining unit further configured to determine the first log file based on the multiple log file copies; the first log file including: a file portion that is recorded in all of the multiple log file copies, and a differential file portion included in each of the multiple log file copies respectively.

[0019] In some implementations, an obtaining unit is configured to obtain a first log file from the first cluster after detecting a failure of the first cluster, including: the obtaining unit is configured to, after detecting a failure of the first cluster, read log files recorded in multiple nodes in the first cluster to obtain multiple log file copies; the obtaining unit is further configured to display, in an operation interface, a difference file part included in each of the multiple log file copies; the obtaining unit is further configured to determine a first log file according to a first operation of a user on the operation interface; the first operation is used to indicate selecting a target difference file part from the difference file parts included in each of the log file copies; the first log file includes: a file part recorded in all of the multiple log file copies, and the target difference file part.

[0020] In some implementations, the first cluster and the second cluster are deployed in different container management clusters; or, the first cluster and the second cluster are deployed in different physical machines in the same container management cluster.

[0021] In some implementations, the container management cluster is a kubernetes cluster.

[0022] In some implementations, the first cluster and the second cluster are etcd clusters with the same version.

[0023] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory and a processor, where the memory is configured to store a computer program, and the processor is configured to, when executing the computer program, enable the electronic device to implement the fault handling method of the storage system in any of the above implementations.

[0024] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which, when a computer program is executed by a computing device, enables the computing device to implement the fault handling method of the storage system in any of the above implementations.

[0025] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when running on a computer, enables the computer to implement the fault handling method of the storage system in any of the above implementations.

[0026] In the fault handling method of the storage system provided by the embodiments of the present application, on the one hand, the method pre-creates the second cluster and synchronizes the first cluster and the second cluster periodically by periodically obtaining the snapshot file of the first cluster and loading the snapshot file into the second cluster. In this way, when the first cluster fails, since the snapshot file of the first cluster has been loaded into the second cluster at this time, the second cluster can replace the first cluster to undertake the business in a short time, thereby reducing the Recovery Time Objective (RTO). On the other hand, in this method, when the first cluster fails, the log file (referred to as the first log file) can be obtained from the failed first cluster first, and the part of the first log file that has not been loaded into the second cluster (i.e., the first incremental file) can be loaded into the second cluster, so that the state of the data in the second cluster is closer to the state of the data in the first cluster when the failure occurs, thereby reducing the Recovery Point Objective (RPO). Further, in this method, the first log file is compared with the snapshot file of the first cluster in the most recent period to quickly determine the part of the first log file that has not been loaded into the second cluster (i.e., the first incremental file), thereby further reducing the RTO. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings herein are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present application and used together with the specification to explain the principles of the present application.

[0028] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0029] Figure 1 It is a schematic structural diagram of an etcd cluster provided by an embodiment of the present application;

[0030] Figure 2 It is a schematic structural diagram of a k8s cluster provided by an embodiment of the present application;

[0031] Figure 3 It is a schematic diagram of the product form of a fault handling device provided by an embodiment of the present application;

[0032] Figure 4 It is one of the schematic flowcharts of a fault handling method provided by an embodiment of the present application;

[0033] Figure 5The second schematic flowchart of a fault handling method provided by an embodiment of the present application;

[0034] Figure 6 The third schematic flowchart of a fault handling method provided by an embodiment of the present application;

[0035] Figure 7 The first schematic structural diagram of a fault handling device provided by an embodiment of the present application;

[0036] Figure 8 The second schematic structural diagram of a fault handling device provided by an embodiment of the present application. Detailed implementation manners

[0037] In order to be able to more clearly understand the above objects, features and advantages of the present application, the solution of the present application will be further described below. It should be noted that, without conflict, the embodiments of the present application and the features in the embodiments may be combined with each other.

[0038] In the following description, many specific details are set forth in order to fully understand the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only a part of the embodiments of the present application, rather than all the embodiments.

[0039] First, the related technologies involved in the embodiments of the present application are introduced:

[0040] A storage system refers to a system used to provide data storage services.

[0041] Common storage systems include distributed key-value storage systems. Among them, in a distributed key-value storage system, key-value type data storage is adopted, and the data in its internal multiple storage nodes is consistent.

[0042] For example, common distributed key-value storage systems include etcd clusters and zookeeper clusters. Among them, the name "etcd" is derived from the " / etc" folder of the Uniplexed Information and Computing System (Unix) and the "d" (i.e., distributed) system. The " / etc" folder is the location for storing the configuration data of a single system, while etcd is a large-scale distributed system. Therefore, adding "d" after " / etc" forms "etcd". The literal Chinese translation of zookeeper is "zoo keeper", but in the context of computer science, it is usually retained in its original English name or abbreviated as ZK. Zookeeper is an open-source distributed coordination service. It provides consistency services for distributed applications, such as configuration management, naming services, distributed synchronization, and group services.

[0043] Exemplarily, such as Figure 1 The figure shows a schematic diagram of the structure of an etcd cluster.

[0044] Among them, the etcd cluster 10 may include: a Hyper Text Transfer Protocol server (HTTP server) 101, and an etcd server 102.

[0045] Among them, the HTTP server 101 is used to receive application programming interface (API) requests issued by clients, as well as requests for synchronization and heartbeat information from other etcd nodes.

[0046] In the etcd server 102, the Raft state machine can be used to perform state transitions according to the received raft messages and call the actions in each state. Among them, the content of the raft data can be written to disk.

[0047] In addition, in the etcd server 102, the Write Ahead Log (WAL) can be used to store the status of all data and the indexes of nodes in memory, and persistent storage is also performed through the WAL. In the WAL, logs are recorded in advance before all data is committed.

[0048] Among them, a snapshot is a status snapshot to prevent excessive data. An entry is the specific log content stored.

[0049] Generally, after receiving a user request, etcd cluster 10 forwards the user request to the Store module in etcd server 102 via HTTP server 101 for specific transaction processing. If node data modification is involved, it is handed over to the Raft module for state change and log recording, and then synchronized to other etcd nodes to confirm data submission. Finally, data submission is performed and synchronization is carried out again.

[0050] Exemplarily, in the case of using an etcd cluster to store the key information of a kubernetes (abbreviated as k8s) cluster, as Figure 2 shown is a schematic structural diagram of a k8s cluster.

[0051] Among them, the k8s cluster 20 includes: a manager (master) 201 and one or more nodes (node) 202.

[0052] Among them, master 201 is used to implement the functions of the control plane of the cluster and is responsible for cluster management. Among them, the API server is the only entry for resource operations, used to receive commands input by users, and provides mechanisms such as authentication, authorization, API registration, and discovery. The Scheduler is responsible for cluster resource scheduling and schedules Pods to corresponding nodes according to a predetermined scheduling policy. The Controller is responsible for maintaining the state of the cluster, such as program deployment arrangement, fault detection, automatic scaling, and rolling updates.

[0053] Node 202 is used to implement the functions of the data plane of the cluster and is specifically responsible for providing a running environment for containers. Among them, kubelet is responsible for maintaining the life cycle of containers, including creating, updating, and destroying containers. kube proxy is responsible for providing service discovery and load balancing within the cluster.

[0054] In addition, kubectl is a command-line tool for the k8s cluster. Through it, the cluster itself can be managed, and containerized applications can be installed and deployed on the cluster.

[0055] In addition, the k8s cluster can store information about various resource objects in the etcd cluster. For example Figure 2 in, the k8s cluster 20 stores information about various resource objects in the etcd cluster 10. Specifically, once the kubernetes environment is started, both master 201 and node 202 will store their own information in the etcd cluster 10.

[0056] In the actual application process, the etcd cluster 10 can run in the k8s cluster 20. For example, the etcd cluster 10 can run in the containers of each node of the k8s cluster 20. In addition, the etcd cluster 10 can also run in other k8s clusters outside the k8s cluster 20.

[0057] The following introduces the technical solutions provided in the embodiments of the present application with reference to examples:

[0058] In the related art, taking the etcd cluster as an example, when the etcd cluster (referred to as the first etcd cluster) fails, another etcd cluster (referred to as the second etcd cluster) will be created first, and then the snapshot file of the first etcd cluster before the failure will be loaded into the second etcd cluster so that the data in the second etcd cluster is the same as the data in the first etcd cluster at the time point corresponding to the above snapshot file. Then the services running on the first etcd cluster are switched to the second etcd cluster to resume the operation of the services.

[0059] Based on the above related art, in the embodiments of the present application, it is considered that: on the one hand, through the above related art, the state of the second etcd cluster can only be restored to the state of the first etcd cluster at the time point corresponding to the snapshot file. If the interval time for executing the snapshot task is 1 hour, the recovery point objective (RPO) of the above related art is 1 hour. That is to say, the RPO time of the above related art is relatively long. On the other hand, in the above related art, after the first etcd cluster fails, the second etcd cluster is created and the obtained snapshot file is loaded on the second etcd cluster. Therefore, it takes a relatively long time from the failure of the first etcd cluster to the switching of the service access point to the second etcd cluster and the resumption of the service operation, that is, the recovery time objective (RTO) time of the above related art is relatively long.

[0060] Therefore, the embodiments of the present application provide a method for handling faults in a storage system. On the one hand, in this method, by periodically obtaining the snapshot file of the first cluster and loading the snapshot file into the second cluster, when the first cluster is working properly, the second cluster can be pre-created and the first cluster and the second cluster can be synchronized periodically. In this way, when the first cluster fails, since the snapshot file of the first cluster has been loaded into the second cluster at this time, the second cluster can replace the first cluster to undertake the business in a relatively short time, thereby reducing the RTO. On the other hand, in this method, when the first cluster fails, the log file (referred to as the first log file) can be obtained from the failed first cluster first, and the part of the first log file that has not been loaded into the second cluster (i.e., the first incremental file) can be loaded into the second cluster, so that the state of the data in the second cluster is closer to the state of the data in the first cluster when the failure occurs, thereby reducing the RPO. Further, in this method, by comparing the first log file with the snapshot file of the first cluster in the most recent period, the part of the first log file that has not been loaded into the second cluster (i.e., the first incremental file) can be quickly determined, thereby further reducing the RTO.

[0061] Among them, the execution subject of the method provided by the embodiments of the present application can be a fault handling device. When the fault handling device runs, it can be used to execute all or part of the steps in the method provided by the embodiments of the present application.

[0062] In some implementation manners, as Figure 3 shown in (a) of, the function of the fault handling device can be implemented by software and / or hardware independent of the first cluster (i.e., the failed storage system) and the second cluster (i.e., the storage system that undertakes the business of the first cluster). In other implementation processes, as Figure 3 shown in (b) of, the fault handling device can also run in the second cluster. At this time, the fault handling device can be used as a functional model of software and / or hardware in the second cluster. For the specific form of the fault handling device, no limitation is made in the embodiments of the present application.

[0063] Specifically, as Figure 4 shown, the method provided by the embodiments of the present application may include the following steps:

[0064] S401. The fault handling device periodically obtains the snapshot file of the first cluster and loads the snapshot file into the second cluster.

[0065] Among them, the first cluster and the second cluster are storage systems of the same type. Specifically, the first cluster and the second cluster can be storage systems with one or more of the same functions, frameworks, or versions, so that the business can run normally after switching the business from the first cluster to the second cluster.

[0066] Specifically, the first cluster and the second cluster can be distributed key-value storage systems of the same version. For example, the first cluster and the second cluster can be etcd clusters of the same version. For another example, the first cluster and the second cluster can be zookeeper clusters of the same version.

[0067] For example, on the one hand, a scheduled task can be set in the first cluster so that a snapshot task is executed on the first cluster every preset duration, and snapshot files corresponding to each cycle of the first cluster are obtained. On the other hand, the fault handling device can periodically read the snapshot files corresponding to each cycle of the first cluster according to the preset duration.

[0068] Exemplarily, after obtaining the snapshot file through the snapshot task, the first cluster can store the snapshot file in the public cloud object storage to ensure the highly reliable storage of the snapshot file. Correspondingly, the fault handling device can periodically read the snapshot files corresponding to each cycle of the first cluster from the public cloud.

[0069] It can be understood that the cycle duration for executing the above S401 can be determined according to actual application requirements. For example, the cycle duration can be 1 hour, 6 hours, 12 hours, etc. The present application embodiment does not limit the cycle duration for executing the above S401.

[0070] In this method, the snapshot file of the first cluster can be used to periodically update the second cluster, so that the data in the second cluster at least includes the data in the first cluster at the time point corresponding to the most recent snapshot, thereby reducing the duration of RTO.

[0071] In addition, in some implementation manners, in the embodiments of the present application, the first cluster and the second cluster can be deployed anti-affinity to reduce the probability that the first cluster and the second cluster fail simultaneously.

[0072] On the one hand, the first cluster and the second can be deployed in different container management clusters.

[0073] For example, when both the first cluster and the second cluster are deployed in the k8s form, the first cluster and the second can be deployed in different k8s clusters.

[0074] On the other hand, the first cluster and the second cluster can be deployed in different physical machines in the same container management cluster.

[0075] For example, when both the first cluster and the second cluster are deployed in the k8s form, the first cluster and the second can be deployed in different physical machines in the k8s cluster. Among them, the different physical machines can be physical machines in different racks, physical machines in different computer rooms, etc.

[0076] S402. After the fault handling device detects a fault in the first cluster, it obtains the first log file from the first cluster.

[0077] Among them, the first log file is used to record data operations on the first cluster.

[0078] When the first cluster and the second cluster can be etcd clusters of the same version, the first log file can be a raft file.

[0079] Among them, the fault that occurs in the first cluster in S402 can specifically be various faults that cause the service response of the first cluster to slow down or the first cluster to become unavailable.

[0080] For example, when the first cluster is an etcd cluster, the faults in S402 above can specifically include any one of the fault types shown in the following table:

[0081] Table 1

[0082]

[0083] Among them, in the first log file, it can specifically include the raft logs corresponding to each data operation on the first cluster. Among them, the raft logs include: the key and value of the written data, and information such as the operation content (such as put, update, insert, or delete, etc.).

[0084] S403. The fault handling device compares the first log file with the snapshot file (hereinafter simply referred to as backup file BF1) of the first cluster in the most recent period to obtain the first incremental file that has increased in the first log file compared to backup file BF1.

[0085] Exemplarily, if the first log file includes: log entry 1, log entry 2, log entry 3, log entry 4, and log entry 5. In addition, backup file BF1 includes: log entry 1 and log entry 2. Then the first incremental file includes: log entry 3, log entry 4, and log entry 5.

[0086] S404. The fault handling device loads the first incremental file into the second cluster.

[0087] In the above method, by pre-creating a second cluster and loading backup files in the second cluster, when loading the first log file into the second cluster, only the first incremental file with a smaller data volume needs to be loaded into the second cluster, thereby reducing the time of RTO. In the above possible implementation, it is considered that: in the case of periodically updating the second cluster by the content of the above S401 (that is, periodically obtaining the snapshot file of the first cluster and loading the snapshot file into the second cluster), the snapshot file obtained in the most recent period can be used as the log file in the first cluster that has been loaded into the second cluster to compare the first log file with the backup file. In this way, it is possible to more conveniently determine the part of the first log file that has not been loaded into the second cluster (that is, the first incremental file).

[0088] S405. The fault handling device switches the access point of the target service from the first cluster to the second cluster.

[0089] Among them, the target service includes the services running in the first cluster before the first cluster fails.

[0090] In the above method of the embodiment of the present application, on the one hand, the method periodically obtains the snapshot file of the first cluster and loads the snapshot file into the second cluster, so that when the first cluster is working properly, the second cluster can be pre-created and the first cluster and the second cluster can be synchronized periodically. In this way, when the first cluster fails, since the snapshot file of the first cluster has been loaded into the second cluster at this time, the second cluster can replace the first cluster to undertake the business in a short time, thereby reducing the Recovery Time Objective (RTO). On the other hand, in the method, when the first cluster fails, by first obtaining the log file (referred to as the first log file) from the failed first cluster and loading the part of the first log file that has not been loaded into the second cluster (that is, the first incremental file) into the second cluster, the state of the data in the second cluster can be made closer to the state of the data in the first cluster when the failure occurs, thereby reducing the Recovery Point Objective (RPO). Further, in the method, by comparing the first log file with the snapshot file of the first cluster in the most recent period, the part of the first log file that has not been loaded into the second cluster (that is, the first incremental file) is quickly determined, thereby further reducing the RTO.

[0091] In addition, considering that during the process of executing S402 to obtain the first log file from the first cluster, it is possible to obtain multiple different log file copies from multiple nodes of the first cluster. At this time, it is necessary to process the multiple log file copies in order to obtain the first log file that needs to be loaded into the second cluster. The following introduces the specific implementation process of S402 in three implementation methods:

[0092] As Figure 5 shown, in the first implementation method, S402 may specifically include:

[0093] S402a1. After the failure handling device detects a failure in the first cluster, it reads the log files recorded in multiple nodes in the first cluster to obtain multiple log file copies.

[0094] Among them, the multiple log file copies may respectively include raft files obtained from different nodes.

[0095] S402a2. The failure handling device determines the first log file according to the multiple log file copies.

[0096] Among them, the first log file includes the file parts that are recorded in all the multiple log file copies.

[0097] Exemplarily, the failure handling device obtains three log file copies through S402a1: log file copy a, log file copy b, and log file copy c.

[0098] Among them, log file copy a includes log entry 1, log entry 2, log entry 3, log entry 4, and log entry 5. Each log entry can be used to record information such as the key, value, and operation content corresponding to a user operation. In addition, log file copy b includes log entry 1, log entry 2, and log entry 3. Log file copy c includes log entry 1, log entry 2, log entry 3, and log entry 4.

[0099] It can be seen that the file parts recorded in all three log file copies include: log entry 1, log entry 2, and log entry 3. Therefore, the first log file obtained through S402a2 includes log entry 1, log entry 2, and log entry 3.

[0100] In other words, through the content of S40a1 - S402a2, the different file parts in the multiple log file copies that are different from other log file copies can be deleted, and the file parts that are recorded in all the remaining multiple log file copies are used as the first log file. In this way, the probability of data consistency failure in the second cluster after loading the first log file can be reduced.

[0101] In the second implementation manner, S402 may specifically include:

[0102] S402b1. After the fault handling device detects a fault in the first cluster, it reads the log files recorded in multiple nodes in the first cluster to obtain multiple log file copies.

[0103] Among them, the multiple log file copies may respectively include raft files obtained from different nodes.

[0104] S402b2. The fault handling device determines the first log file according to the multiple log file copies.

[0105] Among them, the first log file includes: the file part recorded in all the multiple log file copies, and the differential file parts respectively included in each log file copy.

[0106] Exemplarily, still taking the log file copy a, log file copy b, and log file copy c in the above text as an example. Among them, the log file copy a includes log entry 1, log entry 2, log entry 3, log entry 4, and log entry 5. The log file copy b includes log entry 1, log entry 2, and log entry 3. The log file copy c includes log entry 1, log entry 2, log entry 3, and log entry 4.

[0107] After the fault handling device obtains the above three log file copies through S402b1, since the log file copy a includes the differential file parts: log entry 4 and log entry 5, and in addition, the log file copy c includes the differential file part: log entry 4. In addition, the file parts recorded in all the three log file copies include: log entry 1, log entry 2, and log entry 3.

[0108] Therefore, through S402b2, the obtained first log file includes: log entry 1, log entry 2, log entry 3, log entry 4, and log entry 5.

[0109] In other words, through the content of S402b1 - S402b2, the differential file parts in each log file copy can be retained and used as the first log file. In this way, as much data stored in the first cluster as possible can be retained, making the data in the second cluster more complete for the rapid recovery of services.

[0110] In the third implementation manner, S402 may specifically include:

[0111] S402c1. After the fault handling device detects a fault in the first cluster, it reads the log files recorded in multiple nodes in the first cluster to obtain multiple log file copies.

[0112] Among them, multiple copies of the log files can respectively include raft files obtained from different nodes.

[0113] S402c2. The fault handling device displays the differential file parts respectively included in each log file copy among the multiple log file copies on the operation interface.

[0114] For example, the fault handling device may include a display, or the fault handling device may be connected to a display in a wired or wireless manner. Furthermore, the fault handling device can display the differential file parts respectively included in each log file copy among the multiple log file copies on the operation interface of the display.

[0115] Exemplarily, still taking the log file copy a, log file copy b, and log file copy c in the above text as an example.

[0116] After the fault handling device obtains the log file copy a, log file copy b, and log file copy c, it can determine the differential file parts respectively included in each log file copy. Among them, the differential file parts included in the log file copy a are: log entry 4 and log entry 5, and the differential file part included in the log file copy c is: log entry 4. Furthermore, the fault handling device can display the differential file part (log entry 4 and log entry 5) included in the log file copy a and the differential file part (log entry 4) included in the log file copy c on the operation interface in the display.

[0117] S402c3. The fault handling device determines a first log file according to the first operation of the user on the operation interface.

[0118] Among them, the first operation is used to indicate selecting a target differential file part from the differential file parts respectively included in each log file copy.

[0119] The first log file includes: the file parts recorded in all multiple log file copies, and the target differential file part.

[0120] For example, after the fault handling device can display the differential file part (log entry 4 and log entry 5) included in the log file copy a and the differential file part (log entry 4) included in the log file copy c on the operation interface in the display, the user (such as a system operation and maintenance personnel) can, through the first operation on the operation interface (for example, the first operation can be operations such as clicking and dragging on the target differential file part), select the target differential file part from log entry 4 and log entry 5. Assuming that the target differential file part includes log entry 4, the fault handling device determines that the first log file includes: log entry 1, log entry 2, log entry 3, and log entry 4.

[0121] In other words, based on the content of S402c1 - S402c3, the content of the first log file can be determined according to the user's operation. In this way, the content of the log file loaded into the second cluster can be made more flexible to facilitate the rapid recovery of the service.

[0122] In addition, in some possible designs, loading the snapshot file into the second cluster in S401 above may specifically include:

[0123] S4011. Compare the first snapshot file with the snapshot file to obtain the incremental file (referred to as the second incremental file) added in the first snapshot file compared with the second snapshot file.

[0124] Among them, the first snapshot file is the snapshot file corresponding to the first cluster in the current cycle, and the second snapshot file is the snapshot file in the cycle before the first snapshot file.

[0125] S4012. Load the second incremental file into the second cluster.

[0126] In the above design, by comparing the snapshot files of two adjacent cycles (i.e., the first snapshot file and the second snapshot file), the second incremental file added in the first snapshot file compared with the second snapshot file can be determined. Then, only by writing the second incremental file into the backup file, the effect of loading the first snapshot file into the second cluster can be achieved.

[0127] Next, taking the first cluster and the second cluster as etcd clusters as an example, the execution processes of the first cluster, the second cluster, and the fault handling device will be introduced by way of example when the method provided in the embodiments of the present application is applied. Specifically, as Figure 6 shown, the method may include:

[0128] S501. The first cluster periodically executes a snapshot task to obtain a snapshot file.

[0129] For example, a timing task can be set in the first cluster to call the etcd cluster command to execute the snapshot task every preset time interval to obtain a snapshot file.

[0130] Among them, after each snapshot file is generated by the first cluster, the snapshot file can be stored with high reliability using common and object storage.

[0131] S502. The fault handling device periodically executes a preset process.

[0132] Among them, the preset process includes: obtaining the snapshot file corresponding to the first cluster in the current cycle and loading the snapshot file into the second cluster.

[0133] Among them, for the specific implementation process of S502, reference can be made to the implementation process of S401 in the above text.

[0134] In addition, in the case where the first cluster fails, the method further includes:

[0135] S503. After detecting that the first cluster fails, the fault handling device obtains a first log file from the first cluster.

[0136] The first log file is used to record modification operations on the data in the first cluster. Specifically, the first log file can be a raft file.

[0137] The specific implementation process of S503 can refer to the implementation process of S402 in the foregoing text.

[0138] S504. The fault handling device loads the first log file into the second cluster.

[0139] The specific implementation process of S504 can refer to the implementation processes of S403 - S404 in the foregoing text.

[0140] S505. The fault handling device switches the access point of the service from the first cluster to the second cluster.

[0141] The specific implementation process of S505 can refer to the implementation process of S405 in the foregoing text.

[0142] Based on the same inventive concept, as an implementation of the foregoing method, an embodiment of the present application further provides a fault handling device. This embodiment corresponds to the foregoing method embodiment. For the convenience of reading, details of the foregoing method embodiment will not be repeated one by one in this embodiment. However, it should be clear that the fault handling device in this embodiment can correspondingly implement all the content of the foregoing method embodiment.

[0143] An embodiment of the present application provides a fault handling device for a storage system, Figure 7 As a schematic structural diagram of the fault handling device, as Figure 7 shown, the data processing device 60 includes:

[0144] A loading unit 601, configured to periodically obtain a snapshot file of the first cluster and load the snapshot file into the second cluster; the first cluster and the second cluster are storage systems of the same type.

[0145] An obtaining unit 602, configured to obtain a first log file from the first cluster after detecting that the first cluster fails; the first log file is used to record data operations on the first cluster.

[0146] A comparison unit 603, configured to compare the first log file with a snapshot file of the first cluster in the most recent period, so as to obtain a first incremental file in the first log file that has increased compared with the snapshot file of the first cluster in the most recent period.

[0147] A loading unit 601 is further configured to load the first incremental file into the second cluster.

[0148] A switching unit 604, configured to switch an access point of a target service from the first cluster to the second cluster; the target service includes services that ran in the first cluster before the first cluster failed.

[0149] In some implementation manners, loading the snapshot file into the second cluster includes: comparing a first snapshot file with a second snapshot file to obtain a second incremental file in the first snapshot file that has increased compared with the second snapshot file; where the first snapshot file is a snapshot file corresponding to the first cluster in the current period, and the second snapshot file is a snapshot file in the period before the first snapshot file; and loading the second incremental file into the second cluster.

[0150] In some implementation manners, an obtaining unit 602, configured to obtain a first log file from the first cluster after detecting that the first cluster fails, includes:

[0151] The obtaining unit 602 is configured to, after detecting that the first cluster fails, read log files recorded in multiple nodes in the first cluster to obtain multiple log file copies;

[0152] The obtaining unit 602 is further configured to determine the first log file according to the multiple log file copies; the first log file includes: a file part that is recorded in all of the multiple log file copies.

[0153] In some implementation manners, an obtaining unit 602, configured to obtain a first log file from the first cluster after detecting that the first cluster fails, includes:

[0154] The obtaining unit 602 is configured to, after detecting that the first cluster fails, read log files recorded in multiple nodes in the first cluster to obtain multiple log file copies;

[0155] The obtaining unit 602 is further configured to determine the first log file according to the multiple log file copies; the first log file includes: a file part that is recorded in all of the multiple log file copies, and a differential file part included in each of the multiple log file copies.

[0156] In some implementations, the obtaining unit 602 is configured to obtain a first log file from the first cluster after detecting a failure of the first cluster, including:

[0157] The obtaining unit 602 is configured to, after detecting a failure of the first cluster, read log files recorded in multiple nodes in the first cluster to obtain multiple log file copies;

[0158] The obtaining unit 602 is further configured to display, in an operation interface, a differential file part included in each of the multiple log file copies;

[0159] The obtaining unit 602 is further configured to determine a first log file according to a first operation on the operation interface by a user; the first operation is used to indicate selecting a target differential file part from the differential file parts included in the respective log file copies; the first log file includes: a file part recorded in all of the multiple log file copies, and the target differential file part.

[0160] In some implementations, the first cluster and the second cluster are deployed in different container management clusters; or, the first cluster and the second cluster are deployed in different physical machines in the same container management cluster.

[0161] In some implementations, the container management cluster is a kubernetes cluster.

[0162] In some implementations, the first cluster and the second cluster are etcd clusters with the same version.

[0163] The fault handling device 60 provided in an embodiment of the present application can execute the fault handling method provided in any of the above embodiments, and its implementation principle and technical effects are similar, which will not be elaborated here.

[0164] Based on the same inventive concept, an embodiment of the present application further provides an electronic device. Figure 8 The following is a schematic structural diagram of the electronic device provided in an embodiment of the present application. As Figure 8 shown, the electronic device provided in this embodiment includes: a memory 701 and a processor 702. The memory 701 is used to store a computer program, and the processor 702 is used to execute the fault handling method of any storage system provided in the above embodiments when executing the computer program.

[0165] Based on the same inventive concept, an embodiment of the present application further provides a computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the computing device is enabled to implement the fault handling method of any storage system provided in the above embodiments.

[0166] Based on the same inventive concept, an embodiment of the present application also provides a computer program product. When the computer program product runs on a computer, it enables the computing device to implement the fault handling method of any of the storage systems provided in the above embodiments.

[0167] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0168] The processor may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0169] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM), and / or non-volatile memory in the form of, for example, read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0170] Computer-readable media include both permanent and non-permanent, removable and non-removable storage media. The storage media can implement information storage by any method or technology, and the information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information accessible by a computing device. As defined herein, computer-readable media do not include transitory media such as modulated data signals and carrier waves.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than limiting them. Although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for handling a storage system failure, characterized in that: The method comprises: Periodically obtaining a snapshot file of a first cluster and loading the snapshot file to a second cluster; the first cluster and the second cluster are storage systems of the same type; After detecting that the first cluster fails, obtaining a first log file from the first cluster; the first log file is used to record data operations on the first cluster; Compare the first log file with a snapshot file of the first cluster in a most recent period to obtain a first incremental file that is added to the first log file compared to the snapshot file of the first cluster in the most recent period; Loading the first incremental file into the second cluster; Switching an access point of a target service from the first cluster to the second cluster; the target service includes a service running in the first cluster before a failure of the first cluster occurs.

2. The method according to claim 1, characterized in that: The step of loading the snapshot file to the second cluster includes: Compare the first snapshot file with the second snapshot file to obtain a second incremental file added to the first snapshot file compared to the second snapshot file; wherein the first snapshot file is a snapshot file corresponding to the first cluster in the current cycle, and the second snapshot file is a snapshot file in a cycle before the first snapshot file; Load the second incremental file to the second cluster.

3. The method according to any one of claims 1 to 2, characterized in that: After detecting that the first cluster fails, obtaining a first log file from the first cluster includes: After detecting that the first cluster has a fault, reading log files recorded in multiple nodes in the first cluster to obtain multiple log file copies; The first log file is determined according to the multiple log file copies; the first log file includes: a file portion recorded in all the multiple log file copies.

4. The method according to any one of claims 1 to 2, characterized in that: After detecting that the first cluster fails, obtaining a first log file from the first cluster includes: After detecting that the first cluster has a fault, reading log files recorded in multiple nodes in the first cluster to obtain multiple log file copies; The first log file is determined according to the multiple log file copies; the first log file includes: a file part recorded in the multiple log file copies, and a difference file part respectively included in each log file copy.

5. The method according to any one of claims 1-2, characterized in that: After detecting that the first cluster fails, obtaining a first log file from the first cluster includes: After detecting that the first cluster has a fault, reading log files recorded in multiple nodes in the first cluster to obtain multiple log file copies; Displaying the difference file parts respectively included in each of the multiple log file copies in the operation interface; A first log file is determined according to a first operation of the user on the operation interface; the first operation is used to indicate the selection of a target difference file part from the difference file parts respectively included in the various log file copies; the first log file includes: a file part recorded in all of the multiple log file copies, and the target difference file part.

6. The method according to any one of claims 1-2, characterized in that: The first cluster and the second cluster are deployed in different container management clusters; Alternatively, the first cluster and the second cluster are deployed in different physical machines in the same container management cluster.

7. A storage system fault handling device, characterized in that: include: A loading unit, used for periodically acquiring a snapshot file of the first cluster and loading the snapshot file into the second cluster; The first cluster and the second cluster are storage systems of the same type; an acquisition unit, configured to acquire a first log file from the first cluster after detecting that a failure occurs in the first cluster; The first log file is used to record data operations on the first cluster; A comparing unit, configured to compare the first log file with a snapshot file of the first cluster in a most recent period, to obtain a first incremental file added to the first log file compared with the snapshot file of the first cluster in the most recent period; The loading unit is further used to load the first incremental file into the second cluster; A switching unit is used to switch an access point of a target service from the first cluster to the second cluster; the target service includes a service running in the first cluster before a failure of the first cluster occurs.

8. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store a computer program and the processor is used to enable the electronic device to implement the storage system fault handling method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computing device, the computing device implements the storage system fault handling method according to any one of claims 1 to 6.

10. A computer program product, characterized in that When the computer program product runs on a computer, the computer is enabled to implement the storage system fault handling method according to any one of claims 1 to 6.

Images