Cluster data recovery method, device and equipment

Abstract

The invention provides a cluster data recovery method, device and equipment, and belongs to the technical field of computers.The method comprises the steps that first data which is newly generated before a preset time point is screened out from all target data of a first cluster, and the target data is data which conforms to a binary file format and is used for data recovery processing of the cluster; converting the first data into second data conforming to a text file format, and deleting data conforming to a preset deletion condition from the second data based on a text search technology; converting the second data after data deletion into third data conforming to a binary file format; and based on the third data, performing data recovery processing on the first cluster.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a cluster data recovery method, device and equipment. Background Art

[0002] With the development of cluster storage technology, clustered data storage has been widely used. Taking the Zookeeper cluster as an example, many public cloud vendors use the Zookeeper cluster to cluster data such as the console metadata and application configuration data of various Platform as a Service (PaaS) components to ensure high data availability.

[0003] In the related technology, the cluster will generate target data (such as the transaction log file and snapshot log file of the zookeeper cluster) in the process of responding to user operations to back up its own node data to support subsequent possible data recovery processing. However, the current data recovery processing can only restore the cluster data to a specific time point (i.e., the generation time point of a certain target data) based on the target data, which is difficult to meet the actual business needs of users. Summary of the invention

[0004] In view of the above problems, embodiments of the present application provide a cluster data recovery method, apparatus and device to overcome the above problems or at least partially solve the above problems.

[0005] In a first aspect of an embodiment of the present application, a cluster data recovery method is provided, the method comprising:

[0006] Filtering out the first data that is most recently generated before a preset time point from all target data of the first cluster, wherein the target data is data that conforms to a binary file format and is used for data recovery processing of the cluster;

[0007] Converting the first data into second data conforming to a text file format, and deleting data conforming to a preset deletion condition from the second data based on a text search technology;

[0008] Convert the second data after the data is deleted into third data conforming to a binary file format;

[0009] Based on the third data, data recovery processing is performed on the first cluster.

[0010] As a possible implementation, deleting data meeting a preset deletion condition from the second data based on a text search technology includes:

[0011] Performing a text search on the second data according to the execution time point of the user's erroneous operation to obtain a command record of the user's erroneous operation;

[0012] The time point associated with the command record of the user's erroneous operation is used as the preset deletion condition, and the command record and the data generated after the command record are deleted from the second data.

[0013] As a possible implementation manner, when the first cluster is a zookeeper cluster, the second data after the data is deleted is: a transaction log file that conforms to a text file format after the transaction log is deleted; and the converting the second data after the data is deleted into third data that conforms to a binary file format includes:

[0014] The transaction headers and transaction bodies of each transaction log remaining in the second data after the data is deleted are serialized into byte arrays respectively through the pre-created binary output archive object, and each of the obtained byte arrays is determined as the third data.

[0015] As a possible implementation, the method further includes:

[0016] Filtering out the latest snapshot log file generated before the preset time point from all target data of the first cluster;

[0017] The performing data recovery processing on the first cluster based on the third data includes:

[0018] The third data and the filtered snapshot log files are used to perform data recovery processing on the first cluster.

[0019] As a possible implementation manner, after serializing the transaction header and the transaction body of each transaction log remaining in the second data after the data is deleted into byte arrays, the method further includes:

[0020] Determining a checksum for each of the byte arrays;

[0021] Each of the byte arrays and each of the checksums are persistently stored.

[0022] As a possible implementation manner, when the first cluster is a zookeeper cluster, the method further includes:

[0023] In a remote computer room relative to the computer room where the first cluster is located, deploying a second cluster including a single node and associated with the first cluster according to parameter information of each node in the first cluster, wherein the second cluster is a zookeeper cluster;

[0024] Receiving, through the single node, node data of each node synchronized from the first cluster to the second cluster;

[0025] Through the single node, according to the received node data of each node, the target data of the first cluster is backed up off-site based on a preset backup number under a preset backup path, and the target data includes a transaction log file and a snapshot log file.

[0026] As a possible implementation manner, in a remote computer room relative to the computer room where the first cluster is located, deploying a second cluster including a single node and associated with the first cluster according to parameter information of each node in the first cluster, including:

[0027] Writing observer type node parameter information into the configuration file of the second cluster to deploy a second cluster including a single observer type node in the remote computer room;

[0028] The parameter information of each node in the first cluster is written into the configuration file of the second cluster, and the identification information of the single observer type node is modified according to the parameter information to associate the second cluster with the first cluster.

[0029] As a possible implementation, the method further includes:

[0030] In the event that a failure occurs in the computer room where the first cluster is located, redeploying the first cluster in a target computer room;

[0031] Data recovery processing is performed on the redeployed first cluster according to the target data backed up off-site by the second cluster for the first cluster.

[0032] According to a second aspect of an embodiment of the present application, a cluster data recovery device is provided, the device comprising:

[0033] A first screening module is used to screen out the first data that is most recently generated before a preset time point from all target data of the first cluster, wherein the target data is data that conforms to a binary file format and is used for data recovery processing of the cluster;

[0034] A first conversion module, used for converting the first data into second data conforming to a text file format, and deleting data conforming to a preset deletion condition from the second data based on a text search technology;

[0035] A second conversion module, used for converting the second data after the data is deleted into third data conforming to a binary file format;

[0036] The first recovery module is used to perform data recovery processing on the first cluster based on the third data.

[0037] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the cluster data recovery method described in the first aspect are implemented.

[0038] According to a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the cluster data recovery method described in the first aspect are implemented.

[0039] The embodiments of the present application include the following advantages: by screening out the first data most recently generated before a preset time point, the latest node data backup before the preset time point can be obtained, and then by converting the first data into second data that conforms to a text file format, the second data is parsed and edited with the aid of text search technology, thereby obtaining a node data backup at a data recovery time point actually expected by the user (i.e., the second data after the data is deleted), and then by converting the second data after the data is deleted into third data that conforms to a binary file format, the data of the first cluster can be restored to the data recovery time point actually expected by the user based on the third data, thereby achieving more flexible and personalized recovery of cluster data, and being able to better meet the actual business needs of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0041] Figure 1 It is an implementation flow chart of a cluster data recovery method in an embodiment of the present application;

[0042] Figure 2 It is a schematic diagram of an implementation process of personalized backup and recovery for user misoperation in an embodiment of the present application;

[0043] Figure 3 It is a schematic diagram of an implementation process of transaction log file processing in an embodiment of the present application;

[0044] Figure 4 This is a schematic diagram of a system architecture for off-site backup in an embodiment of the present application;

[0045] Figure 5 This is a schematic diagram of a cluster for off-site backup in an embodiment of the present application;

[0046] Figure 6 It is a schematic diagram of an implementation process of a remote backup in an embodiment of the present application;

[0047] Figure 7 It is a schematic diagram of an implementation process of full data recovery in an embodiment of the present application;

[0048] Figure 8 It is a structural diagram of a cluster data recovery device in an embodiment of the present application;

[0049] Fig. 9 It is a schematic diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0050] To facilitate understanding of the technical solutions provided by the present application, the main technical concepts involved in the embodiments of the present application are briefly described below.

[0051] Zookeeper: Zookeeper is a distributed, open source distributed application coordination service. A cluster built on Zookeeper (i.e., a Zookeeper cluster) is usually composed of multiple nodes, which coordinate with each other and save data status. Among them, there are three types (or roles) of nodes in the Zookeeper cluster: leader, follower, and observer.

[0052] Leader node: It provides read and write services to clients and is responsible for initiating and resolving votes and updating system status.

[0053] Follower node: It provides read services to clients, participates in the leader node election process, and participates in the "more than half of the writes succeed" strategy for write operations: for example, in a cluster deployment, if the leader node sends a transaction request (such as a create instruction) to the majority (for example, more than half) of the follower nodes and the request is successfully executed, the transaction request is considered to have been successfully executed.

[0054] Observer node: It is used to expand the data reading performance of the cluster without affecting the voting and data writing performance of the cluster. Since the observer node does not participate in voting, it is not a key node of the zookeeper cluster. That is to say, even if the observer node is abnormal (for example, unable to synchronize data) or disconnected from the cluster, it will not affect the availability of the cluster.

[0055] ZXID: It is a transaction identifier (IDentification, ID) generated by the leader node, which is an increasing hexadecimal number with a total of 64 bits (corresponding to binary format). ZXID consists of two parts: the upper 32 bits represent the current leader node term number, which is an increasing number; the lower 32 bits are the transaction counter, which is an increasing number; that is, ZXID is increasing as a whole.

[0056] Transaction log file: The Zookeeper cluster persists data in the form of transaction logs in files. That is, when the Zookeeper cluster processes each update request, it must first write the relevant transaction log to the file, and then synchronize the data to the memory database to prevent the loss of relevant memory data (that is, node data) after the node crashes.

[0057] Snapshot log file: The ZooKeeper cluster persists the full memory data at a certain moment in the form of a snapshot log to a file, such as writing it to a specified disk file. By default, every 100,000 transaction logs written to the file trigger a snapshot log file write and roll the transaction log, that is, switch to a new transaction log file for writing.

[0058] It is understandable that the snapshot log file is the stock data (that is, the data that already exists in the cluster), and the transaction log file is the incremental data (that is, the data newly added to the cluster). The sum of the two is the maximum amount of data. Among them, the file name suffix ZXID of the snapshot log file is the transaction ID of the last transaction submitted at the moment when the snapshot log file is triggered to be written; for example, if the transaction with transaction ID ZXID5 triggers the snapshot log file to be written, then the snapshot log file name is snapshot.ZXID5, and the transaction ID of the next transaction after triggering the snapshot log file to be written is ZXID6, then the file name of the transaction log file to be written to the transaction log is log.ZXID6.

[0059] In conjunction with the accompanying drawings, a cluster data recovery method, device and equipment provided by the embodiments of the present application are described in detail through some embodiments and their application scenarios.

[0060] First, refer to Figure 1 FIG. 1 is a flowchart of a cluster data recovery method provided by an embodiment of the present application. The method may include the following steps:

[0061] Step S11: Filter out the first data that is most recently generated before a preset time point from all target data of the first cluster, wherein the target data is data that conforms to a binary file format and is used for data recovery processing of the cluster.

[0062] In specific implementation, the preset time point can be flexibly set by the user according to actual business needs. For example, if the user needs to restore data before 10 o'clock, 10 o'clock can be used as the preset time point to search for target data. When searching, the timestamps of different target data can be compared with the preset time point to obtain the first data generated before 10 o'clock.

[0063] The first data may include one or more types of data in the relevant target data; for example, to ensure data processing efficiency, the first data may only include incremental data (such as transaction log files) in the relevant target data (i.e., full data), thereby filtering out large amounts of stock data (such as snapshot log files) to avoid spending more time processing these stock data. When performing data recovery later, the relevant stock data and the processed incremental data are combined for data recovery processing.

[0064] As a possible implementation, when the first cluster is a zookeeper cluster, the transaction ID of the corresponding transaction can be first found according to the preset time point, and then the first data can be filtered out by comparing the file name suffix of different target data (i.e., transaction log file and snapshot log file) with the found transaction ID.

[0065] Step S12: converting the first data into second data conforming to a text file format, and deleting data conforming to a preset deletion condition from the second data based on a text search technology.

[0066] In a specific implementation, a format conversion tool such as LogFormatter (ie, a log formatting method provided by the zookeeper cluster) may be used to convert the first data from a binary file format to a text file format that can be understood by a user, thereby obtaining the second data.

[0067] Then, based on a text search technique such as a keyword search, data that meets the preset deletion condition is found from the second data and deleted. For example, all data generated after a specific time point is found from the second data and deleted (corresponding to the case where the preset deletion condition is a time-related field value), or specific data and data generated thereafter are found from the second data and deleted (corresponding to the case where the preset deletion condition is a command record-related field value such as an operation type).

[0068] It can be understood that, based on the preset deletion conditions, the data in the second data that the user does not want to recover can be deleted, so that during the data recovery process, the cluster data can be restored to the data recovery time point actually expected by the user rather than to the generation time of the specific target data, thereby realizing customized recovery of the cluster data (i.e., recovery to any time point), which can better meet the actual business needs of the user.

[0069] Step S13: converting the second data after the data is deleted into third data conforming to the binary file format.

[0070] In specific implementation, a formatting method provided by the cluster may be used or a binary output archiving object may be created to convert the second data after the data is deleted into third data conforming to a binary file format.

[0071] Step S14: Based on the third data, perform data recovery processing on the first cluster.

[0072] In a specific implementation, if the third data contains all the data required for data recovery processing (such as full data), the third data can be used to directly perform data recovery processing; if the third data only contains part of the data required for data recovery processing (such as incremental data), the remaining data required for data recovery processing (such as existing data) is obtained, and then the third data and the acquired data are used in combination to perform data recovery processing.

[0073] As a possible implementation mode, when the first cluster is a zookeeper cluster, the transaction log files and snapshot log files most recently generated before the preset time point are screened out from all target data of the first cluster, and the screened transaction log files are processed as the first data to obtain third data. Then, the third data and the screened snapshot log files are used to perform data recovery processing on the first cluster, thereby improving processing efficiency by eliminating the analysis and editing steps of snapshot log files with large data volumes.

[0074] By adopting the technical solution of the embodiment of the present application, by screening out the first data most recently generated before a preset time point, the latest node data backup before the preset time point can be obtained, and then by converting the first data into second data that conforms to a text file format, the second data is parsed and edited with the aid of text search technology, thereby obtaining the node data backup at the data recovery time point actually expected by the user (that is, the second data after the data is deleted), and then by converting the second data after the data is deleted into third data that conforms to a binary file format, the data of the first cluster can be restored to the data recovery time point actually expected by the user based on the third data, thereby achieving more flexible and personalized recovery of cluster data, which can better meet the actual business needs of users.

[0075] As a possible implementation manner, the above-mentioned text search technology is used to delete data that meets the preset deletion condition from the second data, including:

[0076] Step S21: performing a text search on the second data according to the execution time point of the user's erroneous operation to obtain the command record of the user's erroneous operation.

[0077] In a specific implementation, the time-related field value in the second data can be compared with the execution time point based on a text search technology such as a keyword search, so as to find the command record that is newly generated at or after the execution time point in the second data. It can be understood that the command record contains a backup of the data that the user mistakenly operated (such as the data that the user mistakenly modified).

[0078] Step S22: taking the time point associated with the command record of the user's erroneous operation as the preset deletion condition, and deleting the command record and the data generated after the command record from the second data.

[0079] In a specific implementation, the time point associated with the command record of the user's erroneous operation may be the time point when the command record was generated and the time point thereafter, which is used as the preset deletion condition to delete the command record and the data generated after the command record from the second data, that is, to delete the backup of all data after the user's erroneous operation from the second data, so that during data recovery processing, the cluster data can be accurately restored to the correct data before the user's erroneous operation, thereby achieving correct recovery of related services, thereby better meeting the actual business needs of users.

[0080] Take the first cluster as the zookeeper cluster as an example, refer to Figure 2 The schematic diagram of the implementation process of personalized backup and recovery for user misoperation is shown, and the implementation process includes the following steps:

[0081] 1. Determine the time point to be restored (i.e., determine the preset time point).

[0082] 2. Select appropriate snapshot log files and transaction log files.

[0083] In this step, search for all the backed up data in the zookeeper cluster (i.e., all the target data), select the latest snapshot log file generated before that time point and the transaction log file corresponding to the snapshot log file. For example, if you need to restore data before 10 o'clock, search for the latest snapshot log file before 10 o'clock and the transaction log file corresponding to the snapshot log file.

[0084] 3. Format the transaction log file in binary file format (ie, the first data) into text file format.

[0085] In this step, the transaction log file may be opened by LogFormatter to convert the transaction log file in a binary file format into a text file format (ie, second data) that can be understood by a user.

[0086] 4. Find the command records of user's incorrect operations.

[0087] 5. Delete the specified transaction log.

[0088] In this step, the transaction log containing the command record of the user's erroneous operation and all subsequent transaction logs in the transaction log file are deleted.

[0089] 6. Regenerate the transaction log file in binary file format (ie, the third data).

[0090] In this step, a binary file is regenerated based on the transaction log remaining after the deletion in the previous step to be used as a new backup recovery file (ie, a binary transaction log file).

[0091] 7. Personalized backup and recovery.

[0092] In this step, the data of the zookeeper cluster is restored in a personalized manner based on the selected snapshot log file and the newly generated binary transaction log file in the previous step.

[0093] As a possible implementation manner, when the first cluster is a zookeeper cluster, the second data after the data is deleted is: a transaction log file that conforms to a text file format after the transaction log is deleted; and the converting the second data after the data is deleted into third data that conforms to a binary file format includes:

[0094] The transaction headers and transaction bodies of each transaction log remaining in the second data after the data is deleted are serialized into byte arrays respectively through the pre-created binary output archive object, and each of the obtained byte arrays is determined as the third data.

[0095] In the specific implementation, a binary output archive object is created in advance, and then the remaining transaction logs in the transaction log file are traversed, and the transaction header (including information such as ZXID, session ID (sessionId)) and transaction body (including information such as command records of operations such as create (create) and delete (delete)) of each transaction log therein are serialized through the binary output archive object to obtain the byte arrays corresponding to each of the transaction logs (that is, the transaction header and transaction body after binary format conversion), and these byte data are used as the third data to participate in subsequent data recovery processing.

[0096] In this implementation, considering that data in binary file format is required for data recovery processing, and Zookeeper only provides LogFormatter, that is, it only provides a method for formatting binary files into text files that can be read by users, and does not provide a method for converting the formatted text files into binary files. Therefore, the embodiment of the present application introduces a binary output archive object to realize the binary file format conversion of the transaction log file, and specifically converts the format of the key data (i.e., the transaction header and transaction body) in the transaction log file required for data recovery processing to improve data processing efficiency.

[0097] Optionally, after serializing the transaction header and transaction body of each transaction log remaining in the second data after the data is deleted into byte arrays, the method further includes:

[0098] Determining a checksum for each of the byte arrays;

[0099] Each of the byte arrays and each of the checksums are persistently stored.

[0100] In specific implementation, a checksum can be determined for each byte array by using XOR check or the like, and each byte array and each checksum can be persistently stored, for example, each byte array and each checksum can be written to a disk to achieve persistent storage. When performing data recovery processing later, the corresponding byte array and checksum can be read from the disk, and the integrity and accuracy of the read byte array can be verified based on the checksum to ensure the integrity and accuracy of the data used in the data recovery processing.

[0101] Take the first cluster as the zookeeper cluster as an example, refer to Figure 3The schematic diagram of the implementation process of transaction log file processing shown in FIG. 1 includes the following steps:

[0102] 1. Use LogFormatter to read the transaction log file and convert it into a text file format.

[0103] 2. Filter and delete the specified command records.

[0104] In this step, according to the preset deletion conditions (such as time, operation type, session ID, etc.), the transaction logs to be deleted are found from the transaction log file (i.e., the second data) after format conversion and deleted, or the transaction logs that do not meet the preset deletion conditions are saved in a list.

[0105] 3. Create a binary output archive object.

[0106] 4. Serialize the transaction header and transaction body in the transaction log.

[0107] In this step, all remaining transaction logs are traversed and serialized in sequence. Optionally, the transaction header and transaction body in the transaction log are serialized.

[0108] 5. Calculate the checksum based on the byte array generated by serialization to ensure the integrity and accuracy of the transaction log.

[0109] 6. Write the byte array and checksum to a disk file.

[0110] 7. Release system resources.

[0111] As a possible implementation manner, when the first cluster is a zookeeper cluster, the method further includes:

[0112] Step S31: In a remote computer room relative to the computer room where the first cluster is located, deploy a second cluster including a single node and associated with the first cluster according to parameter information of each node in the first cluster, where the second cluster is a zookeeper cluster.

[0113] In specific implementation, refer to Figure 4 As shown in the system architecture diagram of off-site backup, computer rooms A and B can be each other's backup computer rooms (i.e., off-site computer rooms). Each computer room is deployed with a backup and recovery platform for executing the cluster data recovery method provided in the embodiment of the present application to achieve personalized recovery and off-site backup of cluster data.

[0114] For example, the first cluster may be a cluster in computer room A. In this case, a backup and recovery platform in computer room B (i.e., a remote computer room relative to the computer room where the first cluster is located) deploys a second cluster including a single node and associated with the first cluster in computer room B according to parameter information (such as address information, etc.) of each node in the first cluster, so as to realize remote backup of the first cluster.

[0115] Step S32: receiving, through the single node, the node data of each node synchronized from the first cluster to the second cluster.

[0116] In specific implementation, refer to Figure 5 As shown in the cluster diagram of off-site backup, after the first cluster in computer room A is associated with the second cluster in computer room B, the first cluster will synchronize its own node data (such as the data of nodes 1, 2 and 3) to a single node in the second cluster in real time, so that the single node can have the latest node data of the first cluster.

[0117] Step S33: through the single node, according to the received node data of each node, the target data of the first cluster is backed up off-site based on a preset backup number under a preset backup path, the target data including transaction log files and snapshot log files.

[0118] In specific implementation, the single node will generate snapshot log files and transaction log files (i.e., target data) based on the synchronized data, and save the target data in a preset backup path. The number of snapshot log files and transaction log files saved in the preset backup path is determined by the preset backup number, that is, the larger the preset backup number, the larger the time span of the snapshot log files and transaction log files saved in the preset backup path. For example, as the preset backup number increases, the single node can save the snapshot log files and transaction log files of the latest week, the latest month, or the latest year in the preset backup path.

[0119] In this embodiment, considering that the write and read response time to the client is reduced, the zookeeper cluster is usually deployed in a computer room, therefore, in order to reduce the loss caused by the computer room-level failure, it is necessary to perform off-site backup of the snapshot log file and the transaction log file of the zookeeper cluster, to support subsequent possible disaster recovery. But the current off-site backup scheme is all through commands such as sftp, remotely logs in to the host where the relevant files of the zookeeper cluster are located, and the snapshot log and the transaction log are downloaded to the local. This mode is based on the scheduled task to perform full backup each time, if the snapshot log file number is a lot, the host resources of the zookeeper cluster will be consumed, affecting its normal operation, and because the scheduled task has a time interval (i.e., delay), it is impossible to ensure that the data of the off-site backup are all the latest.

[0120] To solve the above problems, the embodiment of the present application deploys a second cluster containing a single node and associated with the first cluster in a remote computer room, so as to utilize the data synchronization characteristics of the zookeeper cluster (i.e., the final consistency of the zookeeper cluster) to synchronize the node data in the first cluster to a single node of the second cluster in real time. The single node then generates a snapshot log file and a transaction log file according to the synchronized data. The snapshot log file and the transaction log file can be regarded as the remote backup of the snapshot log and the transaction log of the first cluster. Thus, the influence of the scheduled task on the real-time performance of the remote backup can be avoided, and the occupation of the host resources of the first cluster by the full backup can be avoided, so as to ensure that the first cluster can work normally. And the implementation of the entire remote backup process does not require any configuration of the original zookeeper cluster (i.e., the first cluster), which is easier to implement.

[0121] Optionally, in a remote computer room relative to the computer room where the first cluster is located, deploying a second cluster including a single node and associated with the first cluster according to parameter information of each node in the first cluster, including:

[0122] Writing observer type node parameter information into the configuration file of the second cluster to deploy a second cluster including a single observer type node in the remote computer room;

[0123] The parameter information of each node in the first cluster is written into the configuration file of the second cluster, and the identification information of the single observer type node is modified according to the parameter information to associate the second cluster with the first cluster.

[0124] In specific implementation, a configuration file of the second cluster can be created and filled in the host of the remote computer room, so that a second cluster including a single node and associated with the first cluster can be deployed in the remote computer room; and the single node can be configured as an observer type in the configuration file of the second cluster, so that by virtue of the characteristics of the observer type node (such as not participating in voting and other data synchronization unrelated to the remote backup), the single node can be dedicated to the remote backup of the first cluster, thereby avoiding the single node from affecting the first cluster (such as voting with it and other data synchronization unrelated to the remote backup), thereby reducing the occupancy of related host resources such as disk space.

[0125] For example, refer to Figure 6 The schematic diagram of the implementation process of off-site backup shown in FIG. 1 mainly includes the following steps:

[0126] 1. Get the parameter information of the zookeeper cluster to be backed up (i.e. the first cluster).

[0127] In this step, the backup and recovery platform pre-establishes a network connection with the relevant zookeeper cluster (such as the first cluster) so as to log in to the corresponding host through the user password of the first cluster, thereby obtaining the parameter information of each node in the configuration file of the first cluster (i.e., the zoo.cfg file). The parameter information may include the Internet Protocol (IP) address, communication port, election port, client port, snapshot log, and transaction log directory of each node.

[0128] 2. Fill in the configuration file of the backup zookeeper cluster (i.e. the second cluster).

[0129] In this step, the backup and recovery platform creates a configuration file for the second cluster, and fills in information such as the system and application to which the second cluster belongs, the IP address of the relevant node, the host user password, and the deployment path in the configuration file.

[0130] Taking the first cluster containing three nodes as an example, the backup and recovery platform fills in the data entries of server.1, server.2 and server.3 (corresponding to the three nodes) in the configuration file of the second cluster based on the parameter information of the three nodes, and fills in the data entry of server.4 (corresponding to a single observer type node in the second cluster): server.4 = host IP: communication port: election port: observer, for example, fill in server.4 = 127.0.0.1: 2888: 3888: observer. In this way, the deployment of the second cluster is realized, and the second cluster is associated with the first cluster, that is, a communication connection is established between the second cluster and the first cluster, and the first cluster synchronizes its own node data to the second cluster in real time.

[0131] Among them, in order to ensure the normal implementation of subsequent data synchronization, when associating the first cluster with the second cluster, it is necessary to modify the identification information of the single observer type node according to the parameter information of each node in the first cluster. For example, the file content of the myid file of the host where the above-mentioned single observer type node is located is modified to 4 to ensure that the identification information of each node in the mutually associated clusters is not repeated.

[0132] 3. After the environment test is passed, off-site backup is performed according to the preset backup path and preset backup number.

[0133] In this step, the host environment is detected, such as detecting whether the ports of the host where the single node is located are connected to the hosts where the nodes in the first cluster are located, etc. After the environment detection passes, the single node is started to perform off-site backup according to the preset backup path and preset backup number.

[0134] Optionally, a mode query command may be used to check whether the single node is normal. If the returned query value is Mode=observer, it may be determined that the single node is normal.

[0135] 4. Display the snapshot log and transaction log backed up in real time under the preset backup path to the user.

[0136] As a possible implementation, the method further includes:

[0137] In the event that a failure occurs in the computer room where the first cluster is located, redeploying the first cluster in the computer room;

[0138] Data recovery processing is performed on the redeployed first cluster according to the target data backed up off-site by the second cluster for the first cluster.

[0139] The target computer room may be the computer room where the first cluster is located (ie, the computer room after the fault is repaired) or a new computer room.

[0140] In this implementation, if all the Zookeeper cluster metadata is lost due to a computer room failure, the backup and recovery platform can restore the metadata of the first cluster to the new computer room in full based on the target data backed up offsite, thereby quickly restoring business.

[0141] For example, refer to Figure 7 The schematic diagram of the implementation process of full data recovery shown in FIG. 1 mainly includes the following steps:

[0142] 1. Get the latest data of the off-site backup of the second cluster.

[0143] 2. Fill in the configuration file of the first cluster.

[0144] In this step, a configuration file of the first cluster is created on the host of the target computer room, and information such as the IP address, port, storage path of the snapshot log and transaction log of the relevant nodes is filled in the configuration file.

[0145] 3. Redeploy the first cluster in the target computer room.

[0146] In this step, a host environment test such as a network connection status may be performed on the target computer room first, and if the host environment test passes, the first cluster may be redeployed in the target computer room.

[0147] 4. Copy the latest data backed up off-site to the relevant hosts of the first cluster.

[0148] In this step, the latest snapshot log file and transaction log file backed up off-site are copied to a designated directory of the host where each node in the first cluster is located, so that the first cluster can perform data recovery in a data synchronization manner.

[0149] 5. Start the first cluster and check the node data recovery status of the first cluster.

[0150] Optionally, after starting the first cluster, a mode query command may be used to check whether each node in the first cluster is normal. If the returned query value is mode=leader or mode=follower, it can be determined that the corresponding node is normal.

[0151] For the method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the order of the actions described, because according to the embodiments of the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0152] Second, Figure 8 : is a schematic diagram of the structure of a cluster data recovery device according to an embodiment of the present application, the device comprising:

[0153] The first screening module 100 is used to screen out the first data that is most recently generated before a preset time point from all target data of the first cluster, wherein the target data is data that conforms to a binary file format and is used for data recovery processing of the cluster;

[0154] A first conversion module 200, configured to convert the first data into second data conforming to a text file format, and to delete data conforming to a preset deletion condition from the second data based on a text search technique;

[0155] A second conversion module 300, used to convert the second data after data deletion into third data conforming to a binary file format;

[0156] The first recovery module 400 is configured to perform data recovery processing on the first cluster based on the third data.

[0157] By adopting the technical solution of the embodiment of the present application, by screening out the first data most recently generated before a preset time point, the latest node data backup before the preset time point can be obtained, and then by converting the first data into second data that conforms to a text file format, the second data is parsed and edited with the aid of text search technology, thereby obtaining the node data backup at the data recovery time point actually expected by the user (that is, the second data after the data is deleted), and then by converting the second data after the data is deleted into third data that conforms to a binary file format, the data of the first cluster can be restored to the data recovery time point actually expected by the user based on the third data, thereby achieving more flexible and personalized recovery of cluster data, which can better meet the actual business needs of users.

[0158] Optionally, the first conversion module 200 includes:

[0159] A text search module 201, configured to perform a text search on the second data according to the execution time point of the user's erroneous operation, to obtain a command record of the user's erroneous operation;

[0160] The record deletion module 202 is used to use the time point associated with the command record of the user's erroneous operation as the preset deletion condition to delete the command record and the data generated after the command record from the second data.

[0161] Optionally, when the first cluster is a zookeeper cluster, the second data after the data is deleted is: a transaction log file conforming to a text file format after the transaction log is deleted;

[0162] The second conversion module 300 includes:

[0163] The first conversion submodule 301 is used to serialize the transaction header and transaction body of each transaction log remaining in the second data after the data is deleted into byte arrays through a pre-created binary output archive object, and determine the obtained byte arrays as the third data.

[0164] Optionally, the device further comprises:

[0165] A second screening module 500 is used to screen out the latest snapshot log file generated before the preset time point from all target data of the first cluster;

[0166] The first recovery module 400 includes:

[0167] The first recovery submodule 401 is configured to perform data recovery processing on the first cluster using the third data and the filtered snapshot log files.

[0168] Optionally, the device further comprises:

[0169] A first processing module 600 is used to serialize the transaction header and transaction body of each transaction log remaining in the second data after the data is deleted into byte arrays, and then determine a checksum for each byte array;

[0170] The second processing module 700 is used to persistently store each of the byte arrays and each of the checksums.

[0171] Optionally, the device further comprises:

[0172] The fourth processing module 800 is configured to, when the first cluster is a zookeeper cluster, perform the following steps:

[0173] In a remote computer room relative to the computer room where the first cluster is located, deploying a second cluster including a single node and associated with the first cluster according to parameter information of each node in the first cluster, wherein the second cluster is a zookeeper cluster;

[0174] Receiving, through the single node, node data of each node synchronized from the first cluster to the second cluster;

[0175] Through the single node, according to the received node data of each node, the target data of the first cluster is backed up off-site based on a preset backup number under a preset backup path, and the target data includes a transaction log file and a snapshot log file.

[0176] Optionally, the fourth processing module 800 is further configured to perform the following steps:

[0177] In a remote computer room relative to the computer room where the first cluster is located, deploying a second cluster including a single node and associated with the first cluster according to parameter information of each node in the first cluster, including:

[0178] Writing observer type node parameter information into the configuration file of the second cluster to deploy a second cluster including a single observer type node in the remote computer room;

[0179] The parameter information of each node in the first cluster is written into the configuration file of the second cluster, and the identification information of the single observer type node is modified according to the parameter information to associate the second cluster with the first cluster.

[0180] Optionally, the device further comprises:

[0181] A first deployment module 901 is used to redeploy the first cluster in a target computer room when a failure occurs in the computer room where the first cluster is located;

[0182] The second recovery module 902 is used to perform data recovery processing on the redeployed first cluster according to the target data backed up off-site by the second cluster for the first cluster.

[0183] It should be noted that the device embodiment is similar to the method embodiment, so the description is relatively simple, and the relevant parts can be referred to the method embodiment.

[0184] The present application also provides an electronic device, referring to Fig. 9 , Fig. 9 Schematic diagram of an electronic device proposed in an embodiment of the present application. Fig. 9 As shown, the electronic device 100 includes: a memory 110 and a processor 120. The memory 110 and the processor 120 are connected via a bus communication. A computer program is stored in the memory 110. The computer program can be run on the processor 120 to implement the steps in the cluster data recovery method disclosed in the embodiment of the present application.

[0185] The embodiment of the present application further provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the cluster data recovery method disclosed in the embodiment of the present application is implemented.

[0186] The embodiment of the present application further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the cluster data recovery method disclosed in the embodiment of the present application.

[0187] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0188] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0189] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, systems, devices, storage media, and program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0190] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0191] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0192] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0193] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0194] The above is a detailed introduction to a cluster data recovery method, device and equipment provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A cluster data recovery method, characterized in that: The method comprises: Filtering out the first data that is most recently generated before a preset time point from all target data of the first cluster, wherein the target data is data that conforms to a binary file format and is used for data recovery processing of the cluster; Converting the first data into second data conforming to a text file format, and deleting data conforming to a preset deletion condition from the second data based on a text search technology; Convert the second data after the data is deleted into third data conforming to a binary file format; Based on the third data, data recovery processing is performed on the first cluster.

2. The method according to claim 1, characterized in that Deleting data that meets a preset deletion condition from the second data based on a text search technology includes: Performing a text search on the second data according to the execution time point of the user's erroneous operation to obtain a command record of the user's erroneous operation; The time point associated with the command record of the user's erroneous operation is used as the preset deletion condition, and the command record and the data generated after the command record are deleted from the second data.

3. The method according to claim 1, characterized in that In the case where the first cluster is a zookeeper cluster, the second data after the data is deleted is: a transaction log file conforming to a text file format after the transaction log is deleted; The step of converting the second data after deleting the data into third data conforming to a binary file format includes: The transaction headers and transaction bodies of each transaction log remaining in the second data after the data is deleted are serialized into byte arrays respectively through the pre-created binary output archive object, and each of the obtained byte arrays is determined as the third data.

4. The method according to claim 3, characterized in that The method further comprises: Filtering out the latest snapshot log file generated before the preset time point from all target data of the first cluster; The performing data recovery processing on the first cluster based on the third data includes: The third data and the filtered snapshot log files are used to perform data recovery processing on the first cluster.

5. The method according to claim 3, characterized in that: After serializing the transaction headers and transaction bodies of each transaction log remaining in the second data after the data is deleted into byte arrays, the method further includes: Determining a checksum for each of the byte arrays; Each of the byte arrays and each of the checksums are persistently stored.

6. The method according to any one of claims 1 to 5, characterized in that: In the case where the first cluster is a zookeeper cluster, the method further includes: In a remote computer room relative to the computer room where the first cluster is located, deploying a second cluster including a single node and associated with the first cluster according to parameter information of each node in the first cluster, wherein the second cluster is a zookeeper cluster; Receiving, through the single node, node data of each node synchronized from the first cluster to the second cluster; Through the single node, according to the received node data of each node, the target data of the first cluster is backed up off-site based on a preset backup number under a preset backup path, and the target data includes a transaction log file and a snapshot log file.

7. The method according to claim 6, characterized in that In a remote computer room relative to the computer room where the first cluster is located, deploying a second cluster including a single node and associated with the first cluster according to parameter information of each node in the first cluster, including: Writing observer type node parameter information into the configuration file of the second cluster to deploy a second cluster including a single observer type node in the remote computer room; The parameter information of each node in the first cluster is written into the configuration file of the second cluster, and the identification information of the single observer type node is modified according to the parameter information to associate the second cluster with the first cluster.

8. The method according to claim 6, characterized in that The method further comprises: In the event that a failure occurs in the computer room where the first cluster is located, redeploying the first cluster in a target computer room; Data recovery processing is performed on the redeployed first cluster according to the target data backed up off-site by the second cluster for the first cluster.

9. A cluster data recovery device, characterized in that: The device comprises: A first screening module is used to screen out the first data that is most recently generated before a preset time point from all target data of the first cluster, wherein the target data is data that conforms to a binary file format and is used for data recovery processing of the cluster; A first conversion module, used for converting the first data into second data conforming to a text file format, and deleting data conforming to a preset deletion condition from the second data based on a text search technology; A second conversion module, used for converting the second data after the data is deleted into third data conforming to a binary file format; The first recovery module is used to perform data recovery processing on the first cluster based on the third data.

10. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the cluster data recovery method according to any one of claims 1 to 9.

Images