A method for storing massive small files in HDFS suitable for scenarios with many reads and few writes.

By merging small files and building multi-level indexes in HDFS, the memory consumption and access efficiency issues of HDFS when storing massive amounts of small files are solved. Modification, update and deletion operations of small files are realized, which is suitable for scenarios with more reads and fewer writes, and improves the storage and access efficiency of the system.

CN117891796BActive Publication Date: 2026-11-17HENAN UNIVERSITY
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202311773617.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-21
Publication Date
2026-11-17
Estimated Expiration
2043-12-21

AI Technical Summary

Technical Problem

When storing and accessing massive amounts of small files, HDFS suffers from high NameNode memory usage and low access efficiency. Existing solutions cannot effectively support file modification and update operations in scenarios with more reads and fewer writes.

Method used

Small files are merged into larger data files, and first-level and second-level indexes are built. The indexing technology combines minimum perfect hash and key-value pairs, supporting modification, update and deletion operations on small files. At the same time, the data files are divided into read-only and read-write files, and operations are converted into append operations through redirection.

Benefits of technology

It improves the speed and efficiency of small file access and merging, supports file operations in scenarios with more reads and fewer writes, reduces the NameNode memory load, and optimizes storage utilization.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117891796B_ABST
    Figure CN117891796B_ABST
Patent Text Reader

Abstract

This invention proposes a method for storing massive small files in HDFS, suitable for scenarios with many reads and few writes, to solve the technical problems of high NameNode occupancy and low access efficiency caused by storing small files in existing HDFS systems. The steps of this invention are as follows: Construction phase: Small files are merged into larger data files, and a first-level index and a second-level index are built for each data file; Usage phase: For query operations, the index is queried and the results are returned; for update, delete, and add operations, they are uniformly converted to add operations, realizing update, delete, and add operations for small files. This invention uses a first-level index combining two indexing technologies, which ensures efficient access while assisting in the modification, update, and deletion operations of small files; it significantly improves the speed of small file access and archive construction; it has the functions of updating, deleting, and adding small files with high efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the technical field of distributed, massive small file storage and HDFS distributed file system, and in particular to an HDFS massive small file storage method suitable for scenarios with many reads and few writes. Background Technology

[0002] Distributed technology is a methodology for designing and developing computer systems, aiming to improve system performance, scalability, and fault tolerance by distributing computation and data storage across multiple computing nodes. Massive small file storage refers to the storage needs for handling a large number of small files, such as images, audio, video, and logs. Massive small file storage faces challenges such as low storage space utilization, degraded file system performance, and complex data management. HDFS Distributed File System: HDFS (Hadoop Distributed File System) is an open-source distributed file system used for storing and processing large-scale datasets. It employs a distributed storage approach, storing data shards across multiple computing nodes to improve system fault tolerance and performance. HDFS is highly scalable, capable of handling large amounts of data and adapting to node failures and additions; through redundant data backups and automatic data recovery mechanisms, it ensures data reliability and durability.

[0003] Hadoop, developed by Apache, is a distributed computing framework and an open-source implementation of Google's cloud computing cluster concept. Because it's open-source, it doesn't have high licensing fees, can run on inexpensive hardware, and fully utilizes cluster resources, leading to its widespread adoption. HDFS (Hadoop Distributed File System) is one of the core modules of Hadoop, providing the underlying massive file storage functionality for the entire framework. HDFS uses a master-slave architecture, with the NameNode acting as the master node, responsible for managing the metadata of the entire distributed file system. DataNodes act as slave nodes, responsible for local data storage. However, while HDFS's storage structure is designed for efficient storage of large files, its storage and access efficiency drops significantly when handling massive numbers of small files. These numerous small files generate a large amount of metadata, all stored in the NameNode's memory, causing the system's file storage capacity to reach a bottleneck. Furthermore, the sheer volume of small files leads to a sharp increase in the time overhead of accessing small files, reducing the overall file access efficiency of the system.

[0004] Currently, the main solution to the HDFS small file storage problem is an archive-based solution. This involves merging small files into larger files, indexing the smaller files, and then storing them, providing an access interface to users. There are two main types of these solutions:

[0005] 1. Merge small files into larger files, build an index, and then use an external system (e.g., HBase, MySQL, Redis) to store the index information or data of the smaller files. This approach provides users with a convenient interface for modifying and updating small files, making it suitable for read-heavy, write-light scenarios. However, integrating external systems typically requires system integration and configuration, which can increase the overall system complexity. Compatibility issues between interfaces and data transmission between different systems may require additional development and debugging work, increasing system maintenance costs and risks.

[0006] 2. Indexes built for small files are stored directly in HDFS, eliminating the need for external systems and reducing system complexity. However, due to HDFS's write-once-read-many design, it does not support random writes. HDFS 2.x added append operations to a file, so many solutions only provide interfaces for adding small files, but not for modifying or updating them. This makes many current solutions unsuitable for read-heavy, write-light applications, limiting their application to scenarios like log systems.

[0007] Patent application number 201910242789.4 discloses a method for storing small files in HDFS. It accesses small files in HDFS through a "packaging-index building-upload" approach, with the following specific steps: i. File saving: 1) Upload small files to a relay platform, storing them by date directory, with each file uploaded to the directory corresponding to the current date; 2) Write a script and set a scheduled task to package directories created before the current day into a tar.gz package and upload it to HDFS; 3) Write a gz parsing program to obtain the offset and length of files within the tar.gz package; 4) Design an HBase rowkey, storing the filename and gz path in the rowkey; ii. File query and retrieval: 5) Calculate the HBase rowkey based on the file's date and filename, then retrieve the file's offset and length from HBase; 6) Use the offset and length to call HDFS to read the file. This invention improves the efficiency of accessing small files in HDFS and saves resource consumption. However, the index of the above invention is stored in HBase and uses the method of merging small files, which does not support deletion or modification. If the small file that has been packaged into tar.gz needs to be changed or deleted at some time later, the above patent does not provide a corresponding solution. Summary of the Invention

[0008] To address the technical problems of high NameNode occupancy and low access efficiency caused by storing small files in existing HDFS, this invention proposes a method for storing massive small files in HDFS suitable for scenarios with many reads and few writes. This method can effectively reduce NameNode memory load and improve the efficiency of merging small files. It does not introduce external systems and can provide users with ways to add, delete, and update small files, making it more adaptable to actual needs.

[0009] To achieve the above objectives, the technical solution of this invention is as follows: a method for storing massive small files in HDFS suitable for scenarios with many reads and few writes, comprising the following steps:

[0010] Step 1, Build Phase: Merge small files into larger data files and build first-level and second-level indexes for each data file;

[0011] Step 2, Usage Stage: If it is a query operation, the index will be queried and the results will be returned. If it is an update, delete, or add operation, it will be uniformly converted to add, realizing the update, delete, and add of small files.

[0012] Preferably, the implementation method of the construction phase is as follows:

[0013] (1) Preprocessing: Calculate the number of data files based on the HDFS data block size, and determine the maximum number of small file metadata files stored in the metadata file;

[0014] (2) Small file merging: Divide the set of small files to be merged into multiple sets, create a dedicated thread for each set, and merge the small files assigned to them into the corresponding data files;

[0015] (3) Index building: Receive the offset and length of the data items of the small file and convert them into temporary storage of the small file metadata information until all threads end; distribute all metadata information to different metadata files according to the hash function; build a first-level index by combining the minimum perfect hash and key-value pair indexing techniques, and build a second-level index for each first-level index by using a scalable hash function.

[0016] Preferably, the method for calculating the number of data files based on the total size of all small files and the HDFS data block size is as follows: obtain the size of all files and sum them up; check if the user has set a maximum data file size; if so, obtain the user-set maximum data file size; otherwise, obtain the default HDFS data block size.

[0017] The maximum number of small file metadata entries stored in the metadata file is either rounded up (total size of all small files / HDFS block size) or a user-defined setting.

[0018] The maximum size of metadata files and data files is set by the configuration file, and the number of metadata files is calculated. The metadata information of small files is scattered and stored in different metadata files.

[0019] Preferably, the steps for merging small files are as follows:

[0020] 1) Divide the massive number of small files into multiple sets based on the number of data files, with the total size of each set not exceeding the HDFS data block size;

[0021] 2) Create a thread for each collection to merge small files into a data file concurrently;

[0022] 3) Pass the information of small files to the index building stage.

[0023] Preferably, step 2) is implemented as follows: obtain the number of sets after partitioning; create an equal number of threads according to the number of sets; each thread is responsible for one set, merging the small files in the set into a data file;

[0024] A data file contains multiple smaller files. The metadata file records which data file each smaller file is stored in and its location within the data file. Each smaller file is stored as a data item within the data file, and each data item is of equal size. The content of each smaller file is converted into a data item and written into the data file.

[0025] Preferably, the information of the small file includes a hash value, a data file ID, a starting position, and the size of the data item, which are temporarily stored in memory and await the end of the thread through a function call; the data item includes the hash value of the small file name, the size of the small file content, and the content of the small file;

[0026] The implementation method for converting the content of each small file into a data item and writing it into the data file is as follows: read the small files sequentially; obtain the name, size, and content of the current small file; calculate the hash value based on the file name; encapsulate the hash value, size, and content into an instance; and write the instance into the corresponding data file.

[0027] Preferably, the steps for constructing the first-level index are as follows:

[0028] 11) Collect metadata information for small files generated by each thread;

[0029] 12) Distribute metadata information to different metadata files according to the dispatch function;

[0030] 13) Build a first-level index for each metadata file using a minimal perfect hash function and key-value pairs;

[0031] The steps for constructing a secondary index for each primary index using an extensible hash function are as follows: calculate the global depth value based on the number of primary index files; construct a directory data structure, where the key is the last global depth bit of the small filename hash value in binary form, and the value is the filename of the primary index file.

[0032] Preferably, the metadata information consists of the following fields: filename of the data file, hash value of the minor filename, offset value of the data item, size of the data item, and opcode; the attributes included in the metadata item are: the filename of the minor file stored in which data file, hash value of the minor filename, offset position of the minor file in the data file, size of the minor file stored in the data file, and the operation method by which the metadata item is added to the metadata file.

[0033] The distribution function is a hash function used to distribute metadata information to different metadata files, and:

[0034]

[0035] Where n represents the total number of small files, m represents the maximum number of items in each metadata file, and fnhash is the hash value of the small file name;

[0036] The ID of the data file is calculated using the dispatch function; the ID of the first-level index file and the ID of the metadata file are consistent.

[0037] The value of global depth is obtained by rounding up to (log_2(number of first-level index files)).

[0038] Preferably, updating involves directly calling the add interface that takes two parameters: filename and file content. Deletion involves querying the index based on the filename, and if the file exists, calling the add interface with the filename and an empty value.

[0039] The data converted for the add operation includes: first-level index data, metadata, and small file content data; that is, the small file content includes new small files and small files with modified content added to the correct data file, a new metadata item is constructed, the new metadata item records the relevant information of the small files involved in the operation, and the metadata item is added to the correct metadata file; then the first-level index points to the new metadata item again;

[0040] Each time an update, add, or delete operation is performed, the contents of the first-level index file corresponding to the metadata file are loaded into memory before the operation is performed, and then the results are rewritten to the first-level index file.

[0041] The data files are divided into read-only data files containing multiple data files and read-write data files that exist only once. When the size of the read-write data file is about to exceed the HDFS data block size, it is converted into a read-only data file and a new read-write file is created. The ID of the new read-write file is equal to the ID of the previous read-write file + 1.

[0042] When there are too many small file metadata items, the split operation in the scalable hashing technology is used to split the metadata file that exceeds the HDFS block size into two new metadata files. Redundant metadata files are deleted and redirected. When traversing the metadata items and redistributing them to the two new metadata files, it is checked whether the metadata item is pointed to by the first-level index. If it is not pointed to, it means that it is useless redundant data and is discarded directly. After the traversal is completed, the new metadata files will have metadata items that do not have the same hash value as the small file name. The first-level index of the two metadata files is reconstructed using the minimum perfect hash function.

[0043] Iterate through each metadata item, access the dynamic first-level index based on the hash value of the metadata item record. If it does not exist, it means that it has not been modified or deleted, so keep it; if it exists, get the metadata item pointed to by its first-level index. If the opcode in the metadata item pointed to by the first-level index is "DELETE", discard it. If the data file ID recorded in the metadata item pointed to by the first-level index is greater than the data file ID recorded in the current metadata item, it means that the current metadata item is an old item, so discard it; otherwise, keep it.

[0044] Preferably, the method for redundant deletion of metadata files is as follows: read the data items in the metadata file, create a temporary metadata item list, and for each metadata item and offset: if the operatorCode of the metadata item is equal to "DELETE", discard this metadata item; otherwise, obtain the hash value of the file name from the metadata item, obtain the index item from the dynamic first-level index, and determine whether the offset of the index item is greater than the offset. If the offset of the index item is greater than the offset, return to redetermine the metadata item and offset; otherwise, add the metadata item to the temporary metadata item list; determine whether there is a next metadata item and offset. If so, return to redetermine the metadata item and offset; otherwise, construct a first-level index based on the temporary data item list and write the temporary data item list into the metadata file.

[0045] The method for redundant data file deletion is as follows: Set the current readable and writable data file to read-only, and create a new readable and writable file based on ID + 1; set a variable tmpID = 0, and create a temporary data file tmpID.tmpData based on the variable tmpID; obtain all read-only file IDs and sort them in ascending order by ID; iterate through the array IDS; call the HDFS file reading API based on the ID.data filename, and encapsulate the data read into memory into a data item collection; iterate through each data item; access the dynamic first-level index based on the hash value recorded in the data item record, and if not stored... If the data item is present, it means it has not been modified or deleted, and the data item is written to the tmpID.tmpData file. If the data item exists, the metadata item pointed to by the first-level index is retrieved. If the opcode in the metadata item pointed to by the first-level index is "DELETE", it is discarded. If the data file ID recorded in the metadata item pointed to by the first-level index is greater than the data file ID currently being read, the current data item is an old item and is discarded. Otherwise, it is written to the tmpID.tmpData file. After the IDs in the array IDS have been traversed, the suffix of all files with the .tmpData extension is changed to the .data extension.

[0046] When writing data to tmpID.tmpData, if the total size of the data being written plus the size of the data to be written exceeds the maximum size of the data file, then tmpID = tmpID + 1, and the tmpID.tmpData file is created again.

[0047] Compared with existing technologies, the beneficial effects of this invention are: based on the mechanism that HDFS cannot perform random writes, it enables modification, update, and deletion operations on small files, making it better suited for scenarios with many reads and few writes. Specifically, it includes:

[0048] 1. A first-level index that combines two indexing techniques ensures efficient access while also facilitating the modification, update, and deletion of small files.

[0049] 2. Dividing data files into read-only data files and read-write data files can ensure that each data file does not exceed the HDFS data block size, thereby improving access efficiency, and can also assist in the modification, update and deletion operations of small files.

[0050] 3. The system generates a lot of old data due to the modification, updating and deletion of small files, which reduces the utilization rate of storage. This invention designs a method to delete redundant data to ensure that the system's storage utilization rate does not decrease with system use.

[0051] Compared to previous archiving solutions, this invention significantly improves the speed of accessing small files and the speed of building archives. Compared to previous solutions, this invention has the functions of updating, deleting, and adding small files, and is more efficient. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a flowchart of the present invention.

[0054] Figure 2 This is a flowchart illustrating how the present invention merges small files into a larger data file.

[0055] Figure 3 A flowchart for constructing a first-level index file for this invention.

[0056] Figure 4 A flowchart for constructing a secondary index for this invention.

[0057] Figure 5 This is a structural diagram of the first-level index of this invention.

[0058] Figure 6 This is a structural diagram of the data file of this invention.

[0059] Figure 7 This is a flowchart illustrating the process of removing redundancy from metadata files in this invention.

[0060] Figure 8 This is a flowchart for data file redundancy removal in this invention.

[0061] Figure 9 This is a comparison chart showing the access effects of the present invention.

[0062] Figure 10 This is a comparison chart showing the updated effects of the present invention.

[0063] Figure 11 This is a comparison chart of the storage effects of the present invention. Detailed Implementation

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] like Figure 1 As shown, a method for storing massive amounts of small files in HDFS, suitable for scenarios with many reads and few writes, is described below:

[0066] Step 1, Build Phase: Merge small files into larger data files and build an index for each data file.

[0067] The steps in the build phase are as follows:

[0068] (1) Preprocessing: Calculate the number of data files (formed by merging multiple small files) based on the HDFS data block size, calculate the maximum number of small file metadata that a metadata file can store, and round up (total size of all small files / HDFS data block size) or set the size by the user.

[0069] The number of data files is calculated based on the total size of all small files and the HDFS block size. The sizes of all files are obtained and summed. It is then checked whether the user has set a maximum data file size. If so, the user-defined maximum data file size is obtained; otherwise, the default HDFS block size (128MB in HDFS 2.x versions) is used. The default size is based on the HDFS block size, but users can also customize it using a configuration file. Users can set the maximum size of the metadata file (which stores metadata information for small files) and the data file (which stores the content of small files) through the configuration file. The number of metadata files is then calculated. Since the metadata information for small files is distributed across different metadata files, a maximum size needs to be set for each metadata file. At least one of the following must be present: the number of data files or the number of metadata files.

[0070] (2) Small file merging: The set of small files to be merged is divided into multiple sets, which ensures that the cumulative size of each part does not exceed a predefined threshold. Then, a dedicated thread is created for each set to merge the small files assigned to it into the corresponding data file.

[0071] like Figure 2 The image shown is an overview of merging small files. The steps are as follows:

[0072] 1) Divide the massive number of small files into multiple sets based on the number of data files. Read the file sizes sequentially, sum them up, and group the results closest to the maximum size. Then start the next group. The total size of each set does not exceed the HDFS block size. HDFS will split the file into multiple data blocks, which are distributed across different datanode nodes. When reading a file, requests are sent to multiple datanode nodes, and the total read time depends on the slowest one.

[0073] 2) Create a thread for each set to concurrently merge small files into a data file. Specifically, obtain the number of sets after partitioning; create an equal number of threads based on the number of sets; each thread is responsible for one set, merging the small files in the set into a data file. Multi-threading accelerates the merging efficiency of small files. Merge massive numbers of small files into multiple larger data files. The data files store the specific content of the small files. A data file contains multiple small files, and the metadata file records which data file each small file is stored in and its location within that data file.

[0074] 3) Pass the relevant information of small files to the index building stage.

[0075] The relevant information for small files includes a hash value, a data file ID (which data file it is stored in), a starting position (the location where it is stored in the data file), and a size (the size of the data item, which is ultimately stored as a data item). It is temporarily stored in memory and waits for the thread to finish before being accessed via a function call.

[0076] The content of each small file is not directly written to the data file. Instead, it is converted into data items and written to the data file. The specific implementation method is as follows: read the small files sequentially; obtain the name, size, and content of the current small file; calculate the hash value based on the filename, encapsulate the hash value, size, and content into an instance, and write this instance to the corresponding data file (if the thread performing this operation is 1, it will write to the data file with ID 1). Each small file is stored in the data file as a data item, and each data item has the same size. The data item contains the following attributes: the hash value of the small file name, the size of the small file content, and the content of the small file.

[0077] (3) Index Construction: During the small file construction phase, the offsets and lengths of data items are received and converted into small file metadata information, which is then temporarily stored. This information is encapsulated into an instance and temporarily stored in memory until all threads terminate. Afterward, index construction distributes all metadata information to different metadata files according to a hash function.

[0078] A first-level index combining two indexing techniques is used: minimum perfect hash and key-value pairs. Minimum perfect hash has a small space requirement (O(n log w) bits) and a time complexity of O(1). This means that for small files requiring querying, the filename hash value can be used as the key, requiring only O(1) complexity to locate the metadata in the metadata file. However, its drawback is that the number of elements in the mapping set must be explicitly defined. For scenarios with frequent reads and infrequent writes, minimum perfect hash can significantly reduce space usage while maintaining O(1) query efficiency. Key-value pairs (e.g., the JAVA HashMap class) have the advantage of O(1) query time complexity, do not depend on the specific number of elements in the mapping set, and can add and delete elements in real time. Their disadvantage is that they consume more space. Therefore, they are suitable for scenarios with infrequent writes.

[0079] like Figure 3 The diagram shown is an overview of building a first-level index, and its steps are as follows:

[0080] 1. Collect metadata information for small files generated by each thread.

[0081] Metadata information consists of the following fields: data file name, small file name hash value, data item offset, data item size, and opcode. For each character in the string, its Unicode encoding is calculated. For the i-th character of the string, its Unicode encoding is multiplied by 31^(n-1), where n is the string length and i is the position of the i-th character (counted from left to right, starting from 0). The contribution of each character is summed to obtain the final hash value. The starting position of the data item to be written is determined by obtaining the current length of the data file before appending. The size of the data file before and after writing the data item is obtained; the difference between the two is the size of the data item. The opcode is obtained based on the user's API call. If the user calls the storage API, the opcode is "STORE", and so on. The metadata item includes the following attributes: the data file name in which the small file is stored, the small file name hash value, the small file offset in the data file, the size of the small file stored in the data file, and the operation method (add, delete, update) used to add the metadata item to the metadata file.

[0082] 2. Distribute metadata information to different metadata files according to the dispatch function.

[0083] The distribution function is a hash function used to distribute metadata information to different metadata files, and its formula is as follows:

[0084]

[0085] Here, n represents the total number of small files, m represents the maximum number of items in each metadata file (obtained by dividing the maximum size of the metadata file by the metadata size; this is a fixed size, and all metadata files have the same size), and fnhash is the hash value of the small filename. Using the above distribution function, the ID of the data file can be calculated, allowing metadata information to be distributed to the corresponding metadata file.

[0086] 3. Build a first-level index for each metadata file.

[0087] The primary index is constructed using a minimal perfect hash function and key-value pairs. The primary index file ID and the metadata file ID are identical. The distribution function is a hash function from the scalable hashing technique. Since scalable hashing is used to construct the secondary index, the primary index file can be obtained by calculating the ID from the secondary index.

[0088] Building secondary indexes: A scalable hash function is used to build secondary indexes for each primary index, which can locate which metadata file the small file to be queried is in with only O(1) time complexity.

[0089] Secondary indexes combined with scalable hash functions, such as Figure 4 As shown, its construction steps are as follows:

[0090] 1. Calculate the corresponding global depth value based on the number of first-level index files, and obtain it by rounding up to (log_2(number of first-level index files)).

[0091] 2. Construct a directory data structure where the key is the last bit of the globaldepth in the binary form of the hash value of the small filename, and the value is the filename of the first-level index file. The key is used to quickly retrieve the ID of the pointed-to metadata file; the directory bucket obtained based on the key contains both the data file and the metadata file ID.

[0092] Step 2, Usage Phase: Implementing operations such as updating, deleting, and adding small files.

[0093] The usage steps are as follows: (1) The user performs the corresponding operation. (2) If it is a query operation, the index is queried and the result is returned. If it is an update, delete, or add operation, it is uniformly converted to add. Update: Just call the add interface (which contains two parameters: file name and file content). Delete: Query the index based on the file name. If it exists, call the add interface (file name, null value).

[0094] The key to implementing small file append, deletion, and update operations lies in converting these operations into HDFS append operations. The data that needs to be converted for append operations mainly consists of three types: first-level index data, metadata, and small file content data. The general idea is to achieve these three operations through redirection. Specifically, the small file content (including new small files and modified small files) is added to the correct data file. The small file name is used as input, mapped to the corresponding data file using an extensible hash, and then the HDFS file append data API is called to write it. A new metadata item is then constructed, recording information related to the small file involved in this operation, and this metadata item is added to the correct metadata file. The first-level index is then redirected to point to the new metadata item. An instance is created using the data file ID (with the same hash value of the small file name) and its starting position in the data file as parameters. This instance is then written to the corresponding metadata file (the metadata file has the same ID as the data file, only a different suffix), and the write position is recorded. Finally, a dynamic first-level index (e.g., a HashMap in Java, where the key-value pair format is <file name hash value, starting position of the metadata item in the metadata file>) is called for redirection. For example, <10101,54>-><10101,99>.

[0095] Since the first-level index is also stored in HDFS, and HDFS does not support random writes, each update, add, and delete operation requires loading the entire contents of the first-level index file corresponding to the metadata file into memory before performing the operation, and then rewriting the result back into the first-level index file. This efficiency is acceptable when dealing with small files, but loading the entire index information of a metadata file becomes extremely time-consuming when dealing with a large number of small files. Therefore, the first-level index uses a combination of two techniques, such as... Figure 5 As shown, there are static first-level indexes (indexes built using a minimal perfect hash function) and dynamic first-level indexes (indexes built using key-value pairs). During the first-level index construction phase and the split phase where the metadata file size exceeds the HDFS data file size, a minimal perfect hash function is used. Its advantage is that it avoids loading the entire index data into memory, resulting in faster access speeds. However, its disadvantage is that metadata items in the constructed metadata set must be unique and cannot be modified once built; this is called a static first-level index. Another technique uses a key-value pair data structure as the first-level index, which is frequently loaded into memory; this is called a dynamic first-level index. It is used for redirecting metadata information during update, delete, and add operations. Since the system serves a read-heavy, write-light scenario, the number of first-level indexes stored using key-value pair data structures is not large, thus avoiding loading the entire index information of a metadata file.

[0096] Data files divide them into two categories, such as Figure 6 As shown, there are two types of data files: read-only data files, which contain multiple data files, and readable-writable data files, of which only one exists at any given time. When the size of a readable-writable data file is about to exceed the HDFS block size, it is converted into a read-only data file. This method records the type of each data file; therefore, simply changing the type of this data file to read-only is sufficient. A new readable-writable file is then created, with its ID equal to the previous readable-writable file ID plus 1. The filename is then set to ID+1.data, and the HDFS file creation API is called. This method ensures that the data file size will not exceed the HDFS block size, avoiding the need for HDFS to read the complete content of a small file across multiple nodes when accessing it.

[0097] When there are too many small file metadata items, it's crucial to ensure the metadata file doesn't exceed the HDFS block size. Therefore, a splitting operation using scalable hashing technology is applied to split metadata files exceeding the HDFS block size into two new metadata files. Redundant metadata files are then deleted and redirected. When redistributing metadata items to the two new metadata files, it checks if the item is pointed to by a first-level index. If not, it's useless redundant data and can be discarded. Each metadata item is iterated over, and based on its hash value, the dynamic first-level index is accessed. If it doesn't exist, it hasn't been modified or deleted and is retained. If it exists, the metadata item pointed to by its first-level index is retrieved. If the opcode in the metadata item pointed to by the first-level index is "DELETE," it's discarded. If the data file ID recorded in the metadata item pointed to by the first-level index is greater than the data file ID recorded in the current metadata item, the current metadata item is an old item and is discarded; otherwise, it's retained. After the iteration, the new metadata files will contain metadata items with no duplicate filename hash values. At this point, a minimal perfect hash function is used to rebuild the first-level index for the two metadata files.

[0098] The system generates a lot of old data due to the modification, updating, and deletion of small files, thus reducing storage utilization. After the system has been running for a certain period, a lot of redundant data will accumulate, referring to old metadata items and old small file contents. Redirecting only achieves logical deletion; the old data still exists in HDFS, so it is necessary to periodically clean up the old data. The flowchart for metadata and data file redundancy removal is as follows: Figure 7 , Figure 8 As shown.

[0099] like Figure 7 The method for redundant deletion of metadata files is as follows:

[0100] The process reads data items from the metadata file, creates a temporary metadata item list, and for each metadata item and offset: if the metadata item's `operateCode` equals "DELETE", discard the metadata item; otherwise, retrieve the hash value of the filename from the metadata item, retrieve the index item from the dynamic first-level index, and check if the offset of the index item is greater than the offset. If the offset of the index item is greater than the offset, return and redetermine the metadata item and offset; otherwise, add the metadata item to the temporary metadata item list. The dynamic first-level index is a key-value pair, where the key is the hash value and the value is the starting position where the metadata item of this file is stored. Therefore, the HDFS file read API is called to read the latest content's metadata item pointed to from the starting position, and the right offset value recorded in the metadata item is retrieved. It is checked whether there is a next metadata item and offset; if so, it returns and redetermines the metadata item and offset; otherwise, the first-level index (i.e., the metadata item and offset) is constructed based on the temporary data item list (the metadata items are temporarily stored in memory, using a list data structure such as Java's ArrayList). Figure 3 (Steps) Write the list of temporary data items to the metadata file, traverse the list, and call the HDFS write file API to write them one by one.

[0101] like Figure 8 As shown, the method for deleting redundant data files is as follows:

[0102] 1. Set the current readable and writable data file to read-only, and create a new readable and writable file based on its ID+1. This step is to avoid conflicts during the redundancy removal process, where the new file would be written to the current readable and writable file. Therefore, the current readable and writable data file is set to read-only, and a new readable and writable file is created.

[0103] 2. Set a variable tmpID = 0, and create a temporary data file tmpID.tmpData based on tmpID. This step stores the small files that can be retained during the redundancy removal process.

[0104] 3. Obtain all read-only file IDs (array IDS[]) and sort them in ascending order by ID.

[0105] 4. Traverse the array IDS.

[0106] 5. Based on the ID.data file name, call the HDFS file reading API to read the data into memory and encapsulate it into a collection of data items.

[0107] 6. Iterate through each data item.

[0108] 7. Based on the hash value of the data item record, access the dynamic first-level index. If it does not exist, it means that it has not been modified or deleted. Write its data item into the tmpID.tmpData file.

[0109] 8. If it exists, retrieve the metadata item pointed to by its first-level index. If the opcode in the metadata item pointed to by the first-level index is "DELETE", discard it.

[0110] 9. If the data file ID recorded in the metadata item pointed to by the first-level index is greater than the data file ID currently being read, it means that the current data item is an old item and should be discarded. Otherwise, it should be written to the tmpID.tmpData file.

[0111] When writing data to tmpID.tmpData, the file size may be exceeded. If the total size of the data being written plus the size of the data to be written exceeds the maximum size of the data file, then tmpID = tmpID + 1, and the tmpID.tmpData file is created again.

[0112] After iterating through all the IDs in the array IDS[], change the extension of all files with the .tmpData extension to .data.

[0113] This invention also employs a small file merging method, but without introducing any external systems. The merged results, metadata, and index information of the small files are directly stored in HDFS. HDFS does not support random writes but does support random reads. HDFS supports overwrite writes (completely overwriting the original content) and append writes (writing new content after the original content without overwriting it). Early versions of HDFS did not support any file update operations; once a file was created, written to, and closed, it could no longer be modified. Hadoop 1.x versions did not support the file append function until Hadoop 2.x versions, where append functionality was added to Hadoop Core, allowing data to be appended to HDFS files. Based on this, storage, deletion, and modification are implemented. The implementation method is as follows: the index architecture design is as follows... Figure 4 and Figure 5 As shown, the data file (small file merge result) architecture design is as follows: Figure 6 As shown.

[0114] The horizontal axis represents the small file dataset (100,000, 200,000, 300,000, and 400,000 respectively). The vertical axis represents time, in milliseconds. Lower time is better. SequenceFile, MapFile, and HAR are all natively provided by HDFS. HPF: refers to the paper: Hadoop Perfect File: A fast and memory-efficient metadata access archive file to face small files problem in HDFS. Proposed Method is the method of this invention. HDFS refers to the native HDFS. Comparison of the invention's performance: Based on the small file dataset on the horizontal axis, 5% of the files were randomly selected and accessed; the time obtained was as follows... Figure 9 As shown, lower time indicates higher efficiency, demonstrating the high access efficiency of this invention. Based on the small file dataset on the horizontal axis, 5% of the files are extracted for updating, and the resulting time is as follows: Figure 10 As shown, lower time means higher efficiency, indicating that the update efficiency of this invention is relatively high. HDFS updates first read the original content into memory, modify it, and then completely overwrite it. Early versions of HDFS did not support any file update operations; once a file was created, its data was written, and it was closed, the file could no longer be modified. Figure 11 As shown, based on the archive file constructed in this invention, a dataset of one thousand files is stored using a storage interface, demonstrating the high storage efficiency of this invention. HDFS also stores one thousand datasets, and the time obtained shows that lower time indicates better efficiency.

[0115] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A HDFS mass small file storage method suitable for a multi-read and less-write scenario, characterized in that, The steps are as follows: Step 1, Build Phase: Merge small files into larger data files and build first-level and second-level indexes for each data file; Step 2, Usage Phase: For query operations, the index is queried and the results are returned. For update, delete, and add operations, they are uniformly converted to add operations to realize update, delete, and add small file operations. The implementation method for the construction phase is as follows: (1) Preprocessing: Calculate the number of data files based on the HDFS data block size, and determine the maximum number of small file metadata files stored in the metadata file; (2) Small file merging: Divide the set of small files to be merged into multiple sets, create a dedicated thread for each set, and merge the small files assigned to them into the corresponding data files; (3) Index building: Receive the offset and length of the data items of the small file and convert them into temporary storage of the small file metadata information until all threads end; Distribute all metadata information to different metadata files according to the hash function: Use a combination of minimum perfect hash and key-value pair indexing techniques to build a first-level index, and use an extensible hash function to build a second-level index for each first-level index; Updates directly call the add interface with two parameters: filename and file content. Deletion queries the index based on the filename. If the file exists, the add interface with the filename and an empty value is called. The data converted for the add operation includes: first-level index data, metadata, and small file content data; that is, the small file content includes new small files and small files with modified content added to the correct data file, a new metadata item is constructed, the new metadata item records the relevant information of the small files involved in the operation, and the metadata item is added to the correct metadata file; then the first-level index points to the new metadata item again; Each time an update, add, or delete operation is performed, the contents of the first-level index file corresponding to the metadata file are loaded into memory before the operation is performed, and then the results are rewritten to the first-level index file. The method for calculating the number of data files based on the total size of all small files and the HDFS data block size is as follows: obtain the size of all files and sum them up; check if the user has set a maximum data file size; if so, obtain the user-set maximum data file size; otherwise, obtain the default HDFS data block size. The maximum number of small file metadata files stored in the metadata file is the sum of the sizes of all small files divided by the HDFS block size, rounded up, or a user-defined setting. The maximum size of metadata files and data files is set by the configuration file, and the number of metadata files is calculated. The metadata information of small files is scattered and stored in different metadata files.

2. The HDFS massive small file storage method for scenarios with many reads and few writes as described in claim 1, characterized in that, The steps for merging small files are as follows: 1) Divide the massive number of small files into multiple sets based on the number of data files, with the total size of each set not exceeding the HDFS data block size; 2) Create a thread for each collection to merge small files into a data file concurrently; 3) Pass the information of small files to the index building stage.

3. The HDFS massive small file storage method for scenarios with many reads and few writes as described in claim 2, characterized in that, The implementation method of step 2) is as follows: obtain the number of sets after partitioning; create an equal number of threads according to the number of sets; each thread is responsible for one set and merges the small files in the set into a data file; A data file contains multiple smaller files. The metadata file records which data file each smaller file is stored in and its location within the data file. Each smaller file is stored as a data item within the data file, and each data item is of equal size. The content of each smaller file is converted into a data item and written into the data file.

4. The HDFS massive small file storage method for scenarios with many reads and few writes as described in claim 3, characterized in that, The information of the small file includes a hash value, a data file ID, a starting position, and the size of the data item. It is temporarily stored in memory and will be called via a function after the thread ends. The data item contains the hash value of the small file name, the size of the small file content, and the content of the small file. The implementation method for converting the content of each small file into a data item and writing it into the data file is as follows: read the small files sequentially; obtain the name, size, and content of the current small file; calculate the hash value based on the file name; encapsulate the hash value, size, and content into an instance; and write the instance into the corresponding data file.

5. The HDFS massive small file storage method for scenarios with many reads and few writes, as described in any one of claims 1-4, is characterized in that, The steps for constructing the first-level index are as follows: 11) Collect metadata information for small files generated by each thread; 12) Distribute metadata information to different metadata files according to the dispatch function; 13) Build a first-level index for each metadata file using a minimal perfect hash function and key-value pairs; The steps for constructing a secondary index for each primary index using an extensible hash function are as follows: calculate the global depth value based on the number of primary index files; construct a directory data structure, where the key is the last global depth bit of the small filename hash value in binary form, and the value is the filename of the primary index file.

6. The HDFS massive small file storage method for scenarios with many reads and few writes as described in claim 5, characterized in that, The metadata information consists of the following fields: filename of the data file, hash value of the small filename, offset value of the data item, size of the data item, and opcode. The attributes included in the metadata item are: the filename of the data file in which the small file is stored, hash value of the small filename, offset position of the small file in the data file, size of the small file stored in the data file, and the operation method by which the metadata item is added to the metadata file. The distribution function is a hash function used to distribute metadata information to different metadata files, and: ; Where n represents the total number of small files, m represents the maximum number of items in each metadata file, and fnhash is the hash value of the small file name; The ID of the data file is calculated using the dispatch function; the ID of the first-level index file and the ID of the metadata file are consistent. The value of global depth is obtained by rounding up.

7. The HDFS massive small file storage method for scenarios with many reads and few writes, as described in claim 1 or 6, is characterized in that... The data files are divided into read-only data files containing multiple data files and read-write data files that exist only once. When the size of the read-write data file is about to exceed the HDFS data block size, it is converted into a read-only data file and a new read-write file is created. The ID of the new read-write file is equal to the ID of the previous read-write file + 1. When there are too many small file metadata items, the split operation in the application scalable hashing technology splits the metadata file that exceeds the HDFS data block size into two new metadata files, and applies redundant metadata file deletion and redirection; When redistributing metadata items to two new metadata files, check if the metadata item is pointed to by the first-level index. If it is not pointed to, it means that it is useless redundant data and is discarded directly. After the traversal is complete, the new metadata file will contain metadata entries with hash values ​​that do not have the same filename. The minimum perfect hash function will be used to rebuild the first-level index of the two metadata files. Iterate through each metadata item, access the dynamic first-level index based on the hash value of the metadata item record, and if it does not exist, it means that it has not been modified or deleted, so keep it; If it exists, retrieve the metadata item pointed to by its first-level index. If the opcode in the metadata item pointed to by the first-level index is "DELETE", discard it. If the data file ID recorded in the metadata item pointed to by the first-level index is greater than the data file ID recorded in the current metadata item, it means that the current metadata item is an old item and discard it; otherwise, keep it.

8. The HDFS massive small file storage method for scenarios with many reads and few writes as described in claim 7, characterized in that, The method for redundant deletion of metadata files is as follows: Read the data items in the metadata file, create a temporary list of metadata items, and for each metadata item and offset: if the operatorCode of the metadata item is equal to "DELETE", discard this metadata item; otherwise, obtain the hash value of the file name from the metadata item, obtain the index item from the dynamic first-level index, and determine whether the offset of the index item is greater than the offset. If the offset of the index item is greater than the offset, return and redetermine the metadata item and offset; otherwise, add the metadata item to the temporary list of metadata items. Determine if there is a next metadata item and offset. If so, return and redetermine the metadata item and offset. Otherwise, build a first-level index based on the temporary data item list and write the temporary data item list to the metadata file. The method for deleting redundant data files is as follows: set the current readable and writable data file to read-only, and create a new readable and writable file based on ID+1; Set a variable tmpID = 0, and create a temporary data file tmpID.tmpData based on the variable tmpID; get all read-only file IDs and sort them in ascending order; iterate through the array IDS; The HDFS file reading API is called based on the ID.data file name to read the data into memory and encapsulate it into a collection of data items; each data item is traversed; based on the hash value of the data item record, the dynamic first-level index is accessed. If it does not exist, it means that it has not been modified or deleted, and the data item is written to the tmpID.tmpData file. If it exists, retrieve the metadata item pointed to by the first-level index. If the opcode in the metadata item pointed to by the first-level index is "DELETE", discard it. If the data file ID recorded in the metadata item pointed to by the first-level index is greater than the data file ID currently being read, then the current data item is an old item and discard it; otherwise, write it to the tmpID.tmpData file. After all the IDs in the array IDS have been traversed, change the extension of all files with the .tmpData extension to .data. When writing data to tmpID.tmpData, if the total size of the data being written plus the size of the data to be written exceeds the maximum size of the data file, then tmpID = tmpID + 1, and the tmpID.tmpData file is created again.

Citation Information

Patent Citations

  • Method and device for storing small file in HDFS

    CN110069451A