Hadoop distributed file system (HDFS)-based storage strategy setting acceleration method for files in batches

Abstract

The invention discloses a batch setting storage strategy acceleration method based on an HDFS (Hadoop Distributed File System), which belongs to the field of big data storage, and comprises the following steps: initiating an RPC (Remote Procedure Call) request to a NameNode through a Java API (Application Program Interface), and applying for a setBatchedStoragePolicy (batch setting storage strategy) file; a setBatchedStoragePolicy method is provided for the Client, the Client can access NameNode metadata in batches, the number of RPC requests is reduced to N / 1000 times from original N times, and the number of times of interaction between the Client and the NameNode is greatly reduced; the number of write Lock is reduced from original N times to N / 1000 times, the lock competition problem and the influence on other operation requests are greatly reduced, when the Client sets storage strategies for a large number of files, NameNode metadata are accessed in batches through a setBatchedStoragePolicy method, the performance of setting the file storage strategies is greatly improved, for example, the performance is improved by more than 200 times when the storage strategies are set for 10W files, and the file storage strategy setting efficiency is greatly improved. The consumed time is about 7 seconds; according to the method, the NameNode RPC pressure can be effectively relieved in a high-concurrency scene, the write Lock lock competition is reduced, and the file setting storage efficiency of the Client is improved.

Description

Technical Field

[0001] The present invention belongs to the field of big data storage, and in particular relates to a method for accelerating storage strategy for batch setting of files based on HDFS. Background Art

[0002] Hadoop Distributed File System (HDFS) is a Master / Backup / Slave system. A Hadoop cluster consists of two NameNodes and several DataNodes. NameNode is a management role responsible for the operation of the file system's namespace. It is responsible for maintaining the mapping of file paths to data blocks, and the mapping of data blocks to DataNodes. DataNode is responsible for processing the read and write requests of the client. It creates, deletes, and replicates data blocks under the unified scheduling of NameNode. HDFS supports heterogeneous storage, which can play their respective advantages according to the different read and write characteristics of each storage medium. Currently, there are 4 supported storage types: RAM_DISK (memory), SSD (SSD disk), DISK (DISK disk), ARCHIVE, and 7 storage strategies are supported, such as ALL_SSD, HOT, etc.

[0003] The client can obtain and set storage policies for files or directories, and provides a Shell command line-like method. In addition, HDFS also provides a Java API as a client programming interface for applications to access the file system.

[0004] The client calls the Java API to access the cluster and perform the setStoragePolicy operation.

[0005] For the existing file (setStoragePolicy) operation, only storage policy can be set for one file or directory. Each storage policy setting operation requires interaction with the NameNode, generating an RPC request. When storage policy needs to be set for multiple files or directories, multiple interactions with the NameNode will occur, causing a sudden increase in the number of NameNode RPC requests. In a high-concurrency scenario, when setting storage policy for the current HDFS, NameNode RPC requests increase sharply, and the Client needs to frequently access NameNode metadata. A large number of network requests result in a large delay in NameNode response.

[0006] In some scenarios, it is necessary to set storage policies for a large number of files, which will affect the performance of NameNode. Therefore, in order to solve the problem that in high-concurrency scenarios, a large number of client requests lead to a sharp increase in NameNode RPC requests and writeLock requests, which makes it time-consuming for the client to set storage policy files (directories) and affects other client operations. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a method for accelerating the storage policy setting of files in batches based on HDFS in view of the shortcomings of the background technology. By providing a method for setting storage policy of files in batches for the Client, the number of interactions between the Client and the NameNode is greatly reduced, thereby reducing the network delay overhead of multiple initiations. At the same time, the CPU overhead of the NameNode is also reduced, and the number of writeLocks is reduced.

[0008] The present invention adopts the following technical solutions to solve the above technical problems:

[0009] A storage strategy acceleration method for batch setting of files based on HDFS, including a client and a name node NameNode;

[0010] Client: an application used to set HDFS file storage policy; HDFS is the Hadoop distributed file system;

[0011] NameNode: The manager of the distributed file system, responsible for managing the file system's namespace, directory structure, metadata information, and providing a backup mechanism;

[0012] Based on the understanding of HDFS principles and role situations, the specific steps are as follows:

[0013] Step 1: The client calls the setBatchedStoragePolicy method of the FileSystem class through the Java API to pass in a list of files or directories and a storage policy. FileSystem provides all interfaces for HDFS file operations. setBatchedStoragePolicy is a method for setting storage policies in batches.

[0014] Step 2: The Client initiates an RPC request to the NameNode to request to set the storage policy for the input file or directory list. RPC is a remote procedure call.

[0015] Step 3: The FSNameSystem class setBatchedStoragePolicy method on the NameNode side verifies whether the requested path is valid and whether it is in safe mode. After holding writeLock, it sets the storage policy for the file or directory. FSNameSystem stores and manages the metadata of the HDFS file operating system, and writeLock is a write lock.

[0016] As a further preferred solution of the method for accelerating the storage policy setting for files in batches based on HDFS of the present invention, in step 1, an RPC request for reading files is initiated to NameNode, and the incoming source file directory and storage policy form are supported, which specifically includes the following steps:

[0017] Step 1.1, the client calls the Java API interface through the get method, which is used to obtain a FileSystem object to continue operating the file operating system;

[0018] Step 1.2, calling the setBatchedStoragePolicy method of the FileSystem class of the present invention, which supports passing in a path list to batch set storage policies for multiple files or directories;

[0019] Step 1.3, convert the directory list into an HDFS absolute path by traversing.

[0020] As a further preferred solution of the method for accelerating the storage policy setting of files in batches based on HDFS of the present invention, in step 3, after receiving the storage policy setting operation request from the Client, NameNode obtains the writeLock to set the storage policy operation for the file, and the specific steps include:

[0021] Step 2.1: NameNode receives the RPC request from Client, which contains the file directory list. The number of requests is reduced from N to 1.

[0022] Step 2.2, after obtaining the writeLock, set the storage policy for each element in the list by traversing the entire file directory list. That is, after obtaining the writeLock, set the storage policy for the files or directories in batches and hold the writeLock once.

[0023] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0024] 1. The present invention proposes a method for accelerating the batch setting of storage policies based on HDFS. The Client sends a constructed list of source files and target files to the NameNode through the Java API to apply for the setBatchedStoragePolicy (batch setting of storage policies) file.

[0025] 2. Provide the setBatchedStoragePolicy method for the Client, so that the Client can access the NameNode metadata method in batches. The number of RPC requests is reduced from N to N / 1000, which greatly reduces the number of interactions between the Client and the NameNode. The number of writeLocks is reduced from N to N / 1000, which greatly reduces the lock contention problem and the impact on other operation requests.

[0026] 3. When the client of the present invention sets storage policies for a large number of files, it accesses NameNode metadata in batches through the setBatchedStoragePolicy method, which greatly improves the performance of setting file storage policies. For example, when setting storage policies for 100,000 files, the performance is improved by more than 200 times, and the time consumption is about 7 seconds;

[0027] 4. The present invention can effectively alleviate NameNode RPC pressure, reduce writeLock lock competition, and improve the storage efficiency of Client file settings in high-concurrency scenarios;

[0028] 5. The present invention provides a method for setting storage policies for files in batches for the Client, thereby greatly reducing the number of interactions between the Client and the NameNode, thereby reducing the network delay overhead of multiple initiations; at the same time, it also reduces the CPU overhead of the NameNode and the number of writeLocks. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0030] Figure 1 It is the overall structure diagram of the present invention;

[0031] Figure 2 It is the number of NN RPC requests before and after setStoragePolicy and setBatchedStoragePolicy of the present invention;

[0032] Figure 3 It is a schematic diagram of the comparison between setStoragePolicy and setBatchedStoragePolicy of the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. The present invention is described in detail below based on the drawings and preferred embodiments, and the purpose and effect of the present invention will become clearer. It should be understood that the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0034] A method for accelerating the storage strategy of files in batches based on HDFS mainly consists of two parts: Client and NameNode. The overall structure is shown in the figure below. Figure 1 shown.

[0035] Client: An application that sets HDFS file storage policies.

[0036] NameNode: The manager of the distributed file system, mainly responsible for managing the file system's namespace, directory structure, metadata information, and providing backup mechanisms.

[0037] Based on the understanding of HDFS principles and roles, we developed a method to accelerate the storage strategy of files in batches based on HDFS. The specific steps are as follows:

[0038] Step 1: The client calls the setBatchedStoragePolicy method of the FileSystem class through the Java API to pass in the file or directory list and storage policy;

[0039] Step 2: The Client initiates an RPC request to the NameNode to request the storage policy for the input file or directory list.

[0040] Step 3: The setBatchedStoragePolicy method of the FSNameSystem class on the NameNode side first verifies whether the requested path is valid and whether it is in safe mode, etc., and sets the storage policy for the file or directory after holding writeLock;

[0041] In step 1, an RPC request to read a file is initiated to the NameNode. The source file (directory) and storage policy format can be passed in. The specific steps are as follows:

[0042] Step 1.1, the client calls the Java API interface through the get method, which is used to obtain a FileSystem object to continue operating the file operating system;

[0043] Step 1.2, calling the setBatchedStoragePolicy method of the FileSystem class of the present invention, which supports passing in a path list to batch set storage policies for multiple files or directories;

[0044] Step 1.3, convert the directory list into an HDFS absolute path by traversing.

[0045] In step 3, after receiving the storage policy setting operation request from the Client, NameNode obtains the writeLock and sets the storage policy operation for the file. The specific steps include:

[0046] Step 2.1: NameNode receives the RPC request from Client, which contains the file (directory) list. The number of requests is reduced from N to 1.

[0047] Step 2.2, after obtaining the writeLock, set the storage policy for each element in the list by traversing the entire file (directory) list. That is, after obtaining the writeLock, set the storage policy for the files or directories in batches, so that the writeLock is only held once.

[0048] In some scenarios, it is necessary to set storage policies for a large number of files, which will affect the performance of NameNode. Therefore, in order to solve the problem that in high-concurrency scenarios, a large number of client requests lead to a sharp increase in NameNode RPC requests and writeLock requests, which makes it time-consuming for the client to set storage policy files (directories) and affects other client operations.

[0049] The present invention proposes a method for accelerating the batch setting of storage policies based on HDFS. The method greatly reduces the number of interactions between the client and the NameNode by providing a method for setting storage policies for files in batches for the client, thereby reducing the network delay overhead of multiple initiations. At the same time, it also reduces the CPU overhead of the NameNode and the number of writeLocks.

[0050] The main contributions of the present invention are as follows:

[0051] The client uses the constructed source and target file lists to initiate an RPC request to the NameNode through the Java API to apply for the setBatchedStoragePolicy (batch storage policy setting) file;

[0052] Provide the client with the setBatchedStoragePolicy method, so that the client can access the NameNode metadata method in batches. The number of RPC requests is reduced from N to N / 1000, greatly reducing the number of interactions between the client and the NameNode. The number of writeLocks is reduced from N to N / 1000, greatly reducing the lock contention problem and the impact on other operation requests.

[0053] When the Client sets storage policies for a large number of files, it accesses the NameNode metadata in batches through the setBatchedStoragePolicy method, which greatly improves the performance of setting file storage policies. For example, when setting storage policies for 100,000 files, the performance is improved by more than 200 times and the time consumed is about 7 seconds.

[0054] The above steps can effectively alleviate the NameNode RPC pressure, reduce writeLock lock competition, and improve the client's file storage efficiency in high-concurrency scenarios.

[0055] Embodiment 1: A method for accelerating storage policy settings in batches based on HDFS, the implementation process specifically includes the following steps:

[0056] 1) The Client obtains a FileSystem object through the FileSystem.get method in Java API mode; calls the setBatchedStoragePolicy method of the FileSystem class of the present invention, which supports the input of a file list to set storage policies for multiple files or directories in batches.

[0057] 2) The Client initiates an RPC request to the NameNode to request the storage policy for the input file or directory list.

[0058] 3) The setBatchedStoragePolicy method of the FSNameSystem class on the NameNode side first verifies whether the requested path is valid and whether it is in safe mode, etc., and sets the storage policy for the file or directory after holding the writeLock; it sets the storage policy for each list element one by one by traversing.

[0059] The specific usage of calling the method provided by the present invention is as follows:

[0060] private Configuration conf=new Configuration();

[0061] FileSystem fs=FileSystem.get(conf);

[0062] #Construct files and directories

[0063] List <path>src = newArrayList<>();

[0064] for(int i=0;i<10000;i++){

[0065] String directory = " / " + i;

[0066]

[0067] Figure 1 The numbers of NN RPC requests before and after setStoragePolicy and setBatchedStoragePolicy, respectively, and the number of files is 10,000. As can be seen from the figure, the number of NN RPC requests increases from N to N / 1000.

[0068] Figure 2 Comparing RpcLockWaitTimeNumOps before and after setStoragePolicy and setBatchedStoragePolicy, the results show that WriteLock is significantly reduced.

[0069] The performance of setStoragePolicy and setBatchedStoragePolicy is tested when the number of files is 10,000. Table 1 shows the performance comparison of setStoragePolicy and setBatchedStoragePolicy.

[0070] Table 1

[0071]

[0072] Those skilled in the art can understand that the above are only preferred examples of the invention and are not intended to limit the invention. Although the invention is described in detail with reference to the above examples, those skilled in the art can still modify the technical solutions recorded in the above examples or replace some of the technical features with equivalents. Any modification, equivalent replacement, etc. made within the spirit and principle of the invention should be included in the protection scope of the invention. All technical features in this embodiment can be freely combined according to actual needs.

[0073] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.< / path>

Claims

1. A method for accelerating the storage strategy of files in batches based on HDFS, characterized in that: It includes two parts: Client and NameNode. Client: an application used to set HDFS file storage policy; HDFS is the Hadoop distributed file system; NameNode: The manager of the distributed file system, responsible for managing the file system's namespace, directory structure, metadata information, and providing a backup mechanism; Based on the understanding of HDFS principles and role situations, the specific steps are as follows: Step 1: The client calls the setBatchedStoragePolicy method of the FileSystem class through the Java API to pass in a list of files or directories and a storage policy. FileSystem provides all interfaces for HDFS file operations. setBatchedStoragePolicy is a method for setting storage policies in batches. Step 2: The Client initiates an RPC request to the NameNode to request to set the storage policy for the input file or directory list. RPC is a remote procedure call. Step 3: On the NameNode side, the FSNameSystem class setBatchedStoragePolicy method verifies whether the requested path is valid and whether it is in safe mode. After holding writeLock, the storage policy is set for the file or directory. FSNameSystem stores and manages the metadata of the HDFS file operating system, and writeLock is a write lock.

2. According to claim 1, a method for accelerating the storage strategy of files in batches based on HDFS is characterized in that: In step 1, an RPC request to read a file is initiated to the NameNode. The source file directory and storage policy format can be passed in. The specific steps are as follows: Step 1.1, the Client calls the Java API interface through the get method, which is used to obtain a FileSystem object to continue operating the file operating system; Step 1.2, calling the setBatchedStoragePolicy method of the FileSystem class of the present invention, which supports passing in a path list to batch set storage policies for multiple files or directories; Step 1.3, convert the directory list into an HDFS absolute path by traversing.

3. According to claim 1, a method for accelerating the storage strategy of files in batches based on HDFS is characterized in that: In step 3, after receiving the storage policy setting operation request from the Client, NameNode obtains the writeLock and sets the storage policy operation for the file. The specific steps include: Step 2.1: NameNode receives the RPC request from Client, which contains the file directory list. The number of requests is reduced from N to 1. Step 2.2, after obtaining the writeLock, set the storage policy for each element in the list by traversing the entire file directory list. That is, after obtaining the writeLock, set the storage policy for the files or directories in batches and hold the writeLock once.

Images