HDFS (Hadoop Distributed File System) data recovery method and system, electronic equipment and computer program product

Abstract

The invention discloses an HDFS (Hadoop Distributed File System) data recovery method and system, electronic equipment and a computer program product, and relates to the field of computers, the HDFS data recovery method comprises the following steps: in response to a data recovery instruction, placing an HDFS service in a security mode, in which data in an HDFS is in a read-only state; at least one initial operation record with deletion operation is inquired in the data change log of the HDFS service, the deletion operation in the initial operation record is replaced with safe operation, the modified data change log is obtained, and the safe operation is operation not related to data change; and generating a snapshot file containing the modified data change log, running the snapshot file, and recovering data corresponding to the at least one initial operation record. According to the method, the deletion operation of the data change log is modified into the safe operation, and the HDFS service is guided to run the file system according to the snapshot file by re-executing the snapshot file which does not involve the deletion operation, so that the mistakenly deleted data is effectively recovered.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an HDFS data recovery method, system, electronic device, and computer program product. Background Art

[0002] As the main underlying storage engine of the big data platform, HDFS (Hadoop Distributed File System) is used to store and process large amounts of data in a large distributed environment. Since HDFS carries important data assets, data security of HDFS is particularly important.

[0003] In the process of using the big data platform, it is inevitable that some users may accidentally delete data due to misoperation or other reasons. Accidental data deletion can be divided into two situations: accidentally deleted data is placed in the recycle bin and accidentally deleted data skips the recycle bin. When accidentally deleted data is placed in the recycle bin, the data can be restored by copying the data and rebuilding the table structure. When accidentally deleted data skips the recycle bin, the data can be retrieved through HDFS snapshots. HDFS snapshots are a technology that copies the state of a specified file system at a certain point in time. The HDFS snapshot file can record complete information about all files and directories in the file system state at a certain point in time.

[0004] However, HDFS snapshots can only restore data at the time of the snapshot. If HDFS does not take a snapshot when data is accidentally deleted, the data will be completely deleted in batches when the accidentally deleted data skips the recycle bin.

[0005] Therefore, how to effectively recover data when accidentally deleted data skips the recycle bin and HDFS does not take a snapshot is a technical problem that needs to be solved urgently. Summary of the invention

[0006] In view of the above problems, the present application provides a method, system, electronic device and computer program product for HDFS data recovery to achieve the purpose of effectively recovering data when the accidentally deleted data skips the recycle bin and HDFS does not take a snapshot. The specific solution is as follows:

[0007] A first aspect of the present application provides a method for recovering HDFS data, the method comprising:

[0008] In response to the data recovery instruction, placing the HDFS service in a safe mode, in which data in the HDFS is in a read-only state;

[0009] Querying at least one initial operation record with a deletion operation in the data change log of the HDFS service, and replacing the deletion operation in the initial operation record with a security operation to obtain a modified data change log, wherein the security operation is an operation that does not involve data change;

[0010] A snapshot file including the modified data change log is generated, and the snapshot file is run to restore data corresponding to the at least one initial operation record.

[0011] In a possible implementation, querying the data change log of the HDFS service for at least one initial operation record with a deletion operation includes:

[0012] Convert the data change log into an XML file, and obtain all operation records in the data change log from the XML file;

[0013] An initial operation record with a delete operation is searched from all the operation records.

[0014] In a possible implementation, the deletion operation in the initial operation record is replaced with a security operation to obtain a modified data change log, including:

[0015] Replacing the operation code of the deletion operation in the initial operation record with the operation code of the security operation;

[0016] Replacing the data code in the deletion operation in the initial operation record with the data code corresponding to the security operation;

[0017] The modified xml file is converted back to the data change log to obtain the modified data change log.

[0018] In a possible implementation, generating a snapshot file including the modified data change log includes:

[0019] Copy the modified data change log to the corresponding data directory of the NameNode in the HDFS service and the corresponding data directory of the JournalNode in the HDFS service respectively;

[0020] Start the NameNode in the recovery mode of the HDFS service, and start the JournalNode and ZooKeeper in the HDFS service, so that the NameNode generates a snapshot file containing the modified data change log after successful startup.

[0021] In a possible implementation, the method further includes:

[0022] Start the Active Namenode node of the Namenode in the HDFS service, and determine the data recovery status through the Active Namenode node monitoring the reporting status of the data block.

[0023] In a possible implementation, the method further includes:

[0024] The snapshot file containing the modified data change log is synchronized to each node in the Namenode.

[0025] A second aspect of the present application provides an HDFS data recovery system, the HDFS data recovery system comprising:

[0026] A security unit, configured to respond to a data recovery instruction and place the HDFS service in a security mode, wherein the data in the HDFS is in a read-only state;

[0027] a replacement unit, configured to query at least one initial operation record with a deletion operation in the data change log of the HDFS service, and replace the deletion operation in the initial operation record with a security operation to obtain a modified data change log, wherein the security operation is an operation that does not involve data change;

[0028] The recovery unit is used to generate a snapshot file including the modified data change log, and run the snapshot file to recover the data corresponding to the at least one initial operation record.

[0029] In a possible implementation, the replacement unit searches the data change log of the HDFS service for at least one initial operation record with a deletion operation, and the specific configuration is as follows:

[0030] Convert the data change log into an XML file, and obtain all operation records in the data change log from the XML file;

[0031] An initial operation record with a delete operation is searched from all the operation records.

[0032] A third aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0033] The memory is used to store computer programs;

[0034] The processor is used to execute the computer program so that the electronic device can implement the HDFS data recovery method of the above-mentioned first aspect or any implementation manner of the first aspect.

[0035] The fourth aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the HDFS data recovery method of the above-mentioned first aspect or any implementation of the first aspect.

[0036] By means of the above technical scheme, the present application provides a method, system, electronic device and computer program product for HDFS data recovery. In response to the data recovery instruction, the method first puts the HDFS service into safe mode to avoid data changes, then replaces the deletion operation of the initial operation record in the data change log with a safe operation to change the data deletion recorded in the data change log, and then generates a snapshot file containing the modified data change log, and runs the snapshot file to restore the data corresponding to at least one initial operation record in the data change log. The method modifies the deletion operation of the data change log into a safe operation to remove the operation of actually deleting the data by mistake, and by re-executing the snapshot file that does not involve the deletion operation, guides the HDFS service to run the file system according to the snapshot file of the data change operation that has been replaced with the safe operation in the snapshot file, so as to achieve the purpose of effectively recovering the deleted data by mistake. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0038] Figure 1 A schematic diagram of a process for completely deleting data in HDFS provided in an embodiment of the present application;

[0039] Figure 2 A flowchart of a HDFS data recovery method provided in an embodiment of the present application;

[0040] Figure 3 A schematic diagram of the structure of an HDFS data recovery system provided in an embodiment of the present application;

[0041] Figure 4 A hardware structure block diagram of an electronic device provided in this application. DETAILED DESCRIPTION

[0042] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0043] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0044] The terms "first", "second" etc. in the specification of the application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable in appropriate circumstances, and this is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0045] As the main underlying storage engine of the big data platform, HDFS (Hadoop Distributed File System) is used to store and process large amounts of data in a large distributed environment. Since HDFS carries important data assets, data security of HDFS is particularly important.

[0046] However, in the process of using the big data platform, it is inevitable that data will be accidentally deleted. After data is accidentally deleted, it can be divided into two situations: accidentally deleted data is placed in the recycle bin and accidentally deleted data skips the recycle bin. When accidentally deleted data is placed in the recycle bin, the data can be restored by copying the data and rebuilding the table structure. When accidentally deleted data skips the recycle bin, the data can be retrieved through HDFS snapshots.

[0047] Since the HDFS snapshot is a copy of the file system based on a time point, it is taken by the user or automatically at regular intervals. Therefore, the HDFS snapshot can only restore the data at the time of the snapshot. If the data is deleted by mistake, HDFS does not take a snapshot, which means that the time when the data was deleted by mistake is between the times of two snapshots. In this case, when the deleted data skips the recycle bin, it cannot be restored through the HDFS snapshot, and the data will be completely deleted.

[0048] The HDFS service involves multiple key components, among which NameNode is responsible for storing metadata information of all files and directories in HDFS. These metadata include file names, directory structures, block distribution of files, replica locations, access permissions, etc. NameNode is also used to process operations on the file system namespace, including creating new files, deleting files, renaming files, creating or deleting directories, etc. In HDFS, files are divided into multiple blocks and distributed on different data nodes (DataNode). NameNode is responsible for managing the distribution and replication of data blocks. DataNode is the node that actually stores data in HDFS. It divides the user's data into multiple data blocks and stores these data blocks in the local file system. Each data block will have multiple copies saved on multiple DataNodes. Even if a DataNode fails, the copies on other DataNodes can still ensure the integrity of the data.

[0049] The process of completely deleting data is as follows: Figure 1 As shown. After the client executes the delete command (Delete), the NameNode (NN) deletes the corresponding file path information in the directory tree, finds the block information of the file to be deleted, puts it into the InvalidateBlocks object, and writes the deletion log in the edit log (data change log). Among them, the InvalidateBlocks object is an internal data structure in the NameNode, which is used to track and manage data block copies that have been marked for deletion but have not been completely removed from the physical storage.

[0050] The NameNode performs heartbeat polling with the DataNode (DN), pulls the block information to be deleted from the InvalidateBlocks object, generates a deletion command (1,000 blocks are deleted each time by default), and sends it to the corresponding DataNode.

[0051] DataNode processes the deletion command according to the NameNode heartbeat command, asynchronously deletes disk data in batches, and reports the deleted blocks to NameNode.

[0052] Therefore, according to the above data deletion process, after the client executes the deletion command, the data is not put into the recycle bin, but is not deleted immediately. Instead, it is asynchronously deleted in batches by NameNode and DataNode. Therefore, when data is deleted by mistake, the HDFS service should be stopped immediately. The sooner it is stopped, the less data loss can be reduced.

[0053] Based on the above asynchronous data deletion process, this embodiment of the present application provides an HDFS data recovery method. The method puts the HDFS service into safe mode, replaces the deletion operation in the data change directory with a safe operation, generates a new snapshot file, and effectively recovers the accidentally deleted data by running the snapshot file.

[0054] The HDFS data recovery method of the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0055] Reference Figure 2 , Figure 2 A flowchart of a method for recovering HDFS data provided by an embodiment of the present application is shown in FIG. Figure 2 As shown, an HDFS data recovery method provided in an embodiment of the present application may include steps S10 to S12, and these steps are described in detail below.

[0056] S10, in response to the data recovery instruction, placing the HDFS service in a safe mode, in which the data in the HDFS is in a read-only state;

[0057] Among them, before data recovery, in order to stop the ongoing asynchronous deletion of data or other data changes in the HDFS service, it is necessary to stop HDFS and related services, such as Zk (distributed coordination service), YARN (resource management center), Hive (data warehouse tool), Impala (interactive SQL query function) and Sentry (authorization service). Furthermore, when stopping the HDFS service, it is necessary to record the Active Namenode node. In HDFS, NameNode is usually configured as two nodes, one node is in Active state (Active Namenode), and the other node is in Standby state (Standby Namenode). The Active Namenode is the Namenode that currently provides services to the outside world.

[0058] In this embodiment, the HDFS service is mainly placed in safe mode, and the HDFS service is not completely stopped. Safe mode is a special state of the HDFS file system. In safe mode, the HDFS service only accepts read data requests, but does not accept change requests such as deleting data or modifying data.

[0059] S11. Query at least one initial operation record with a deletion operation in the data change log of the HDFS service, and replace the deletion operation in the initial operation record with a security operation to obtain a modified data change log. The security operation is an operation that does not involve data changes.

[0060] Among them, the data change log (edit log) is an important log file in HDFS, which is used to record the operation log of the file system. The edit log records every operation of the user on HDFS, including file creation, deletion, modification (such as appending content), permission changes, etc. Since when data is deleted by mistake, the NameNode will write the relevant deletion log in the edit log. Therefore, this embodiment obtains the edit log of the Active Namenode node. In the edit log of the Active Namenode node, the data deletion operation will be recorded. Then, this embodiment can achieve data recovery by modifying the content in the edit log of the Active Namenode node. Specifically, this embodiment first queries the edit log for the initial operation record with the deletion operation, and then replaces the deletion operation in the initial operation record with a safe operation. Safe operations are operations that do not involve data changes, such as setting permission operations, encryption operations, and data backup operations. The specific process of modifying the edit log can be shown in steps one to three as follows:

[0061] Step 1: Convert the data change log into an XML file, and obtain all operation records in the data change log from the XML file;

[0062] Step 2: Query the initial operation record with the delete operation from all operation records.

[0063] Step 3: Replace the operation code of the deletion operation in the initial operation record with the operation code of the security operation;

[0064] Step 4: Replace the data code in the deletion operation in the initial operation record with the data code corresponding to the security operation;

[0065] Step 5: Convert the modified XML file back to the data change log to obtain the modified data change log.

[0066] Among them, this embodiment can view and convert the edit log through the tools provided in the HDFS service (such as Hadoop Offline Edits Viewer), and can also parse the data change log through other programming language programs to convert the non-text format content in the data change log into displayable or understandable content. For example, Java (the main programming language of Hadoop) or other programming languages ​​that support file I / O. When querying the initial operation record with the deletion operation, this embodiment can mainly be based on the operation code (Data Processing Code) in all operation records. Furthermore, in addition to the XML file format, the edit log can also be converted into other file formats that are convenient for content editing, which is not limited in this embodiment.

[0067] Here, an example is provided to illustrate the specific process of modifying the edit log.

[0068] Convert the edit log to an XML file. The execution command can be as follows:

[0069] hdfs oev -i edits_000000000320527467-000000000320527867 -o edits_000000000320527467-000000000320527867.xml

[0070] Find the delete operation (OP_DELETE, the bold code below) and the corresponding file path in the xml file, as shown below:

[0071] <record>

[0072] <opcode> OP_DELETE< / opcode>

[0073] <data><h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr">

[0074] <h2 style=";text-align:left;direction:ltr"> <txid> 320527488< / txid> <h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr">

[0075] <h2 style=";text-align:left;direction:ltr"> <length> 0< / length> <h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr">

[0076] <h2 style=";text-align:left;direction:ltr"> <path> / apps< / path> <h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr">

[0077] <h2 style=";text-align:left;direction:ltr"> <timestamp> 1478404822121< / timestamp> <h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr">

[0078] <h2 style=";text-align:left;direction:ltr"><RPC_CLIENTID> 8a904481-efbe-4e5b-97da-54cda9ca2932< / RPC_CLIENTID><h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr">

[0079] <h2 style=";text-align:left;direction:ltr"><RPC_CALLID> 1< / RPC_CALLID><h2 style=";text-align:left;direction:ltr"> <h2 style=";text-align:left;direction:ltr">

[0080] < / data>

[0081] < / record>

[0082] Edit the xml file and replace OP_DELETE in the delete operation with a security operation. Here we take the permission setting operation OP_SET_PERMISSIONS (the bold code below) as an example

[0083] <record>

[0084] <opcode> OP_SET_PERMISSIONS< / opcode>

[0085] <data>

[0086] <txid> 320527488< / txid>

[0087] <src> / apps< / src>

[0088] <mode> 493< / mode>

[0089] < / data>

[0090] < / record>

[0091] Convert the edited XML file back to edit log. The execution command can be as follows:

[0092] hdfs oev -i edits_000000000320527467-000000000320527867 -o edits_000000000320527467-000000000320527867.xml

[0093] Thus, the modification process of the edit log is completed, and the modified edit log is obtained.

[0094] Among them, in this embodiment, when replacing the deletion operation with a security operation, the operation code of the deletion operation needs to be replaced with the operation code of the security operation, and the data code under the deletion operation also needs to be replaced with the data code corresponding to the security operation to prevent errors when HDFS is running. In the above example, after the deletion operation is replaced with a permission setting operation, since the data corresponding to the data code under the deletion operation cannot be set with permissions, if it is not modified, an error will occur when HDFS is running. Therefore, the data code under the deletion operation needs to be replaced with the data code corresponding to the permission setting operation. As a result, the data corresponding to the deletion operation does not perform the deletion operation, but is converted to a permission setting operation for other data, which can prevent the complete deletion of the data. Of course, in a few cases, it is also possible to only replace the operation code of the deletion operation with the operation code of the security operation without replacing the data code.

[0095] S12. Generate a snapshot file containing the modified data change log, and run the snapshot file to restore data corresponding to at least one initial operation record.

[0096] The snapshot file in this embodiment is the Fsimage file. Fsimage, full name FileSystem Image, is a static snapshot of the HDFS file system at a specific moment. It contains complete information of all files and directories in HDFS, including paths, permissions, timestamps, and locations of data blocks.

[0097] The specific process of generating a snapshot file containing the modified data change log in this embodiment can be shown in the following steps 5 and 6:

[0098] Step 5: Copy the modified data change log to the corresponding data directory of the NameNode in the HDFS service and the corresponding data directory of the JournalNode in the HDFS service;

[0099] Step 6: Start NameNode in the recovery mode of the HDFS service, and start JournalNode and ZooKeeper in the HDFS service, so that after the NameNode is successfully started, it generates a snapshot file containing the modified data change log.

[0100] Among them, JournalNode is a component in HDFS used to share and store the edit log generated by NameNode. NameNode generates a new edit log every once in a while (for example, 2 minutes) and uploads it to JournalNode for storage. When the Active Namenode node (primary node) fails, the Standby Namenode node (backup node) can download the edit log from the JournalNode and merge it with its own fsimage file to generate a new fsimage, thereby taking over the work of the Active Namenode node. ZooKeeper is a distributed, open source coordination service that provides consistency services for distributed applications, and is mainly used to control the active and standby status of the NameNode node.

[0101] Since the edit log of NameNode is stored in JournalNode, after the edit log is modified in this embodiment, it needs to be copied to the corresponding data directory of NameNode and the corresponding data directory of JournalNode respectively to unify the data of NameNode and JournalNode.

[0102] Since the edit log is modified in this embodiment, in order to avoid the situation where the edit log does not correspond to the NameNode when it is started, this embodiment needs to start the NameNode in recovery mode so that the NameNode regenerates an Fsimage file (snapshot file) based on the modified edit log. The recovery mode of the HDFS service is a special startup mode that helps the NameNode to recover itself when the metadata is damaged. In recovery mode, the NameNode will try to skip the damaged editlog records to ensure successful startup, and after successful startup, the NameNode will generate a new Fsimage file and then exit recovery mode.

[0103] In this embodiment, after copying the modified edit log to the corresponding data directory of NameNode and the corresponding data directory of JournalNode, JournalNode and ZooKeeper are started, and NameNode is started in recovery mode. Namenode regenerates an Fsimage file based on the modified edit log copied to the data directory, and the Fsimage file includes the modified edit log.

[0104] Start the Active Namenode of the Namenode in the HDFS service, and monitor the reporting of data blocks through the Active Namenode to determine the data recovery status. When the Active Namenode is started, it will first load the Fsimage file containing the modified edit log, and directly apply the data change operations of the modified edit log to ensure that the metadata information in the memory is the latest.

[0105] After the Active Namenode node successfully loads the modified Fsimage file, it will send a request to the DataNode, asking it to report the data block information, and determine the data recovery status by monitoring the reporting of the data blocks through the Active Namenode node. In this embodiment, the number of reported blocks can be observed through the web UI of the Active Namenode node. The Active Namenode node will automatically exit the safe mode only after 99.99% of the blocks are reported. Therefore, when the Active Namenode node is still in safe mode after all blocks have been reported, it means that some data blocks are still lost and have not been recovered in time, then the Active Namenode node will be exited from safe mode through Cloudera Manager (CM). Furthermore, a partial list of lost files can be displayed on the web UI of the Active Namenode node. If there are too many lost files, you can use some commands to view or obtain the complete list of lost files. The command format can be as follows:

[0106] sudo -u hdfs hdfs fsck / myapps -list-corruptfileblocks>corruptfiles.txt

[0107] This embodiment can delete the metadata information of lost or damaged files in the lost file list. After deletion, real data recovery can be performed by rerunning the lost file information to minimize data loss. The deletion command can be as follows:

[0108] sudo -u hdfs hdfs fsck / myapps -delete> delete_corrupt.log

[0109] In order to avoid data conflicts between the Active Namenode and the Standby Namenode in the Namenode, after the Namenode regenerates a snapshot file, the snapshot file containing the modified data change log needs to be synchronized to each node in the Namenode. Specifically, after the Active Namenode regenerates an Fsimage file containing the modified editlog, the regenerated Fsimage file is copied to the data directory of the Standby Namenode to ensure the data consistency of the active and standby nodes in the Namenode. After the data of the Standby Namenode is unified with the data of the Active Namenode, start the Standby Namenode and exit the safe mode through Cloudera Manager (CM), and other related services in HDFS can also exit the safe mode.

[0110] An HDFS data recovery method provided by an embodiment of the present application, in response to a data recovery instruction, first puts the HDFS service into a safe mode to avoid data changes, then replaces the deletion operation of the initial operation record in the data change log with a safe operation to change the data deletion recorded in the data change log, and then generates a snapshot file containing the modified data change log, and runs the snapshot file to recover the data corresponding to at least one initial operation record in the data change log. This method modifies the deletion operation of the data change log into a safe operation to remove the operation that actually deletes the data by mistake, and by re-executing the snapshot file that does not involve the deletion operation, guides the HDFS service to run the file system according to the snapshot file of the data change operation that has been replaced with the safe operation in the snapshot file, so as to achieve the purpose of effectively recovering the accidentally deleted data.

[0111] Furthermore, the HDFS data recovery method provided in this embodiment cannot ensure the recovery of all accidentally deleted data. The amount of deletion depends on the time when the HDFS service is stopped after the accidentally deleted data. However, through the HDFS data recovery method in this embodiment, the lost file information can be obtained, and the data can be restored as much as possible through historical tracing and re-running, so as to minimize the data loss. Among them, the permission control needs to be well configured, and the user's deletion operation needs to be performed in a standardized manner (for example, without adding -skipTrash) to leave room for data recovery operations.

[0112] The above introduces an HDFS data recovery system provided by an embodiment of the present application. The following will introduce a device for executing the above HDFS data recovery method.

[0113] See also Figure 3 , Figure 3A schematic diagram of the structure of an HDFS data recovery system provided in an embodiment of the present application. Figure 3 As shown, the HDFS data recovery system includes:

[0114] The security unit 100 is used to respond to the data recovery instruction and put the HDFS service into a safe mode. In the safe mode, the data in the HDFS is in a read-only state;

[0115] The replacement unit 110 is used to query at least one initial operation record with a deletion operation in the data change log of the HDFS service, and replace the deletion operation in the initial operation record with a safe operation to obtain a modified data change log, where the safe operation is an operation that does not involve data change;

[0116] The recovery unit 120 is used to generate a snapshot file containing the modified data change log, and run the snapshot file to recover the data corresponding to at least one initial operation record.

[0117] In a possible implementation, the replacement unit 110 searches for an initial operation record with a deletion operation in the data change log of the HDFS service, which may be specifically configured as follows:

[0118] Convert the data change log into an XML file, and obtain all operation records in the data change log from the XML file;

[0119] Query the initial operation record with the delete operation from all operation records.

[0120] In a possible implementation, the replacement unit 110 replaces the deletion operation in the initial operation record with the security operation to obtain the modified data change log, which can be specifically configured as follows:

[0121] Replace the operation code of the delete operation in the initial operation record with the operation code of the security operation;

[0122] Replace the data code in the deletion operation in the initial operation record with the data code corresponding to the safety operation;

[0123] Convert the modified XML file back to the data change log to obtain the modified data change log.

[0124] In a possible implementation, the snapshot file containing the modified data change log is generated in the recovery unit 120, which can be specifically configured as follows:

[0125] Copy the modified data change log to the corresponding data directory of NameNode in the HDFS service and the corresponding data directory of JournalNode in the HDFS service respectively;

[0126] Start the NameNode in the recovery mode of the HDFS service, and start the JournalNode and ZooKeeper in the HDFS service, so that after the NameNode is successfully started, it generates a snapshot file containing the modified data change log.

[0127] In a possible implementation, the HDFS data recovery system further includes a checking unit:

[0128] The inspection unit is used to start the Active Namenode node of the Namenode in the HDFS service, and monitor the reporting status of the data blocks through the Active Namenode node to determine the data recovery status.

[0129] In a possible implementation, the HDFS data recovery system further includes a synchronization unit:

[0130] The synchronization unit is used to synchronize the snapshot file containing the modified data change log to each node in the Namenode.

[0131] The present application also provides an electronic device in an embodiment. Figure 4 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 4 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0132] like Figure 4 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 to a random access memory (RAM) 403. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 403. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0133] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a memory card, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0134] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the HDFS data recovery methods provided in the embodiments of the present application.

[0135] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any HDFS data recovery method provided in an embodiment of the present application.

[0136] It should also be noted that the system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the system embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0137] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0138] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0139] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0140] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0141] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0142] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for recovering HDFS data, characterized in that: The HDFS data recovery method includes: In response to the data recovery instruction, placing the HDFS service in a safe mode, in which data in the HDFS is in a read-only state; Querying at least one initial operation record with a deletion operation in the data change log of the HDFS service, and replacing the deletion operation in the initial operation record with a security operation to obtain a modified data change log, wherein the security operation is an operation that does not involve data change; A snapshot file including the modified data change log is generated, and the snapshot file is run to restore data corresponding to the at least one initial operation record.

2. The HDFS data recovery method according to claim 1, characterized in that: The querying of at least one initial operation record with a deletion operation in the data change log of the HDFS service includes: Convert the data change log into an XML file, and obtain all operation records in the data change log from the XML file; An initial operation record with a delete operation is searched from all the operation records.

3. The HDFS data recovery method according to claim 2, characterized in that: The deletion operation in the initial operation record is replaced with a security operation to obtain a modified data change log, including: Replacing the operation code of the deletion operation in the initial operation record with the operation code of the security operation; Replacing the data code in the deletion operation in the initial operation record with the data code corresponding to the security operation; The modified xml file is converted back to the data change log to obtain the modified data change log.

4. The HDFS data recovery method according to claim 1, characterized in that: The generating of the snapshot file containing the modified data change log comprises: Copy the modified data change log to the corresponding data directory of the NameNode in the HDFS service and the corresponding data directory of the JournalNode in the HDFS service respectively; Start the NameNode in the recovery mode of the HDFS service, and start the JournalNode and ZooKeeper in the HDFS service, so that the NameNode generates a snapshot file containing the modified data change log after successful startup.

5. The HDFS data recovery method according to claim 1, characterized in that: Also includes: The Active Namenode node of the Namenode in the HDFS service is started, and the reporting status of the data blocks is monitored by the Active Namenode node to determine the data recovery status.

6. The HDFS data recovery method according to claim 1, characterized in that: Also includes: The snapshot file containing the modified data change log is synchronized to each node in the Namenode.

7. A HDFS data recovery system, characterized in that: The HDFS data recovery system includes: A security unit, configured to respond to a data recovery instruction and place the HDFS service in a security mode, wherein the data in the HDFS is in a read-only state; a replacement unit, configured to query at least one initial operation record with a deletion operation in the data change log of the HDFS service, and replace the deletion operation in the initial operation record with a security operation to obtain a modified data change log, wherein the security operation is an operation that does not involve data change; The recovery unit is used to generate a snapshot file including the modified data change log, and run the snapshot file to recover the data corresponding to the at least one initial operation record.

8. The HDFS data recovery system according to claim 7, characterized in that: The replacement unit searches the data change log of the HDFS service for at least one initial operation record with a deletion operation, and the specific configuration is: Convert the data change log into an XML file, and obtain all operation records in the data change log from the XML file; An initial operation record with a delete operation is searched from all the operation records.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the HDFS data recovery method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that It includes computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the HDFS data recovery method as described in any one of claims 1 to 6.

Images