A small file processing method based on data pipeline

Abstract

The present application discloses a small file processing method based on a data pipeline, which completes the path parsing, metadata change and data storage operations of the file through a single call, significantly reducing the multiple message interactions in the traditional POSIX interface operation. The data and metadata of the small file are packaged and written to the data pipeline at the same time, avoiding the multiple separate writing of metadata and file data in the traditional system, and reducing the total I / O times. The data pipeline is designed for multi-threaded concurrent writing, which can support multiple clients to operate simultaneously, greatly improving the throughput of the system in the small file processing scenario. The numbering of the data packet makes the log replay sequential and consistent, and can ensure that the metadata and file data in the system are completely synchronized after the write operation is completed. The present application effectively solves the problem of low I / O throughput in the new writing scenario of massive small files mentioned in the background technology, and makes the system more efficient and reliable in practical applications by reducing interactions, reducing I / O and concurrent optimization.

Description

Technical Field

[0001] The present application relates to the technical field of small file processing based on data pipelines, and in particular to a small file processing method based on data pipelines. Background Art

[0002] With the development of information technology and the explosive growth of data volume, distributed file systems have played an important role in modern data storage. Distributed file systems are usually composed of metadata services (MDS), data services (OSD) and clients. Their operations are usually based on standard POSIX interfaces, including file opening, writing, reading and closing. These operations are initiated by the client and involve metadata access and data read and write interactions.

[0003] In the case of massive small files, the performance of existing distributed file systems is often severely limited. Each I / O operation involves a small amount of data, resulting in low overall system throughput. In addition, the metadata query, data interaction, and distributed lock coordination involved in the file operation process all increase system latency and complexity.

[0004] To address this problem, some optimization methods have been proposed in the prior art: metadata appending small file data: the data of small files is directly appended to metadata (such as inode) for storage to reduce the storage and reading overhead of small files. This method is suitable for scenarios with small file sizes, usually processing data less than 4KB. Client data merging: the data of multiple small files are merged on the client side, packaged into a larger file, and then written to the storage device. This method can reduce the number of write operations and improve throughput efficiency.

[0005] However, these optimization methods have significant shortcomings:

[0006] The metadata attachment method will occupy a large amount of metadata space, resulting in a decrease in metadata storage and access performance, especially in scenarios where metadata is frequently operated.

[0007] The client data merging method requires cache storage, which may increase the latency of a single I / O operation. At the same time, both methods cannot effectively reduce the frequent message interaction problems in the scenario of massive small file new writing, such as metadata query and distributed lock coordination.

[0008] In the above context, existing technologies are difficult to meet the needs of efficient storage and access of massive small files, especially in new write scenarios, where file operations involve a large number of message interactions and multiple I / O operations, further reducing system throughput. Therefore, a new optimization method is urgently needed that can significantly improve the efficiency of small file write scenarios while taking into account system performance and compatibility. Summary of the invention

[0009] The present application provides a small file processing method based on a data pipeline, aiming to solve the problem that the existing technology is difficult to meet the needs of efficient storage and access of massive small files.

[0010] A small file processing method based on a data pipeline, the method comprising:

[0011] Provides a customized file writing interface write_new_file to receive the file path and complete data of a small file;

[0012] During the system startup initialization phase, multiple data pipelines are created to store metadata and data of small files. Each data pipeline is a linear space mapped to multiple object storage devices.

[0013] Through the write_new_file interface, the metadata changes and data content of the small file are packaged to generate a data packet containing the file path, data content and metadata change information;

[0014] Assign a unique number to each data packet and write it into the corresponding data pipeline;

[0015] Recording the pipeline ID of the data packet and its position in the pipeline in the metadata of the file;

[0016] In the background of the metadata service, the metadata and file data in the data pipeline are decomposed through log replay and stored in the predetermined storage space respectively.

[0017] In the above solution, optionally, the data pipeline allocation method includes:

[0018] Allocation method based on polling mechanism;

[0019] Mapping allocation method based on file path hash value;

[0020] Dynamic allocation method based on real-time load status;

[0021] The number of data pipelines supports dynamic addition and deletion.

[0022] In the above solution, optionally, the write_new_file interface is executed by the following steps:

[0023] Send the file path and file data to the metadata service;

[0024] The metadata service performs a search operation on the file path. If the path already exists, the corresponding inode is reused. If the path does not exist, a new inode is created.

[0025] Encapsulate the path-related metadata change records and file data content into a data packet.

[0026] In the above solution, optionally, the data pipeline supports multi-threaded concurrent writing, allowing multiple clients to access and write to multiple data pipelines at the same time.

[0027] In the above solution, optionally, the log replay process includes:

[0028] Read data packets sequentially from the data pipeline;

[0029] Extract metadata change records and file data contents according to the serial number sequence of the data packets;

[0030] Writing metadata change information into the metadata storage area;

[0031] Write the file data content to the data storage area.

[0032] In the above solution, optionally, the data pipeline is organized in the form of a log, wherein each data packet contains the following fields:

[0033] Package number;

[0034] File path information;

[0035] File data content;

[0036] File metadata change records;

[0037] Data verification information.

[0038] In the above solution, optionally, before the log replay is completed, the file reading operation is implemented by the following steps:

[0039] The client queries the metadata service for the corresponding metadata based on the file path;

[0040] The data pipe ID and location recorded in the metadata are used to locate the target data packet;

[0041] The client reads the file data content directly from the data pipeline.

[0042] In the above solution, optionally, after the log replay is completed, the file access operation is switched to the access mode of the traditional distributed file system, wherein the metadata and data are stored in the metadata storage area and the data storage area respectively.

[0043] In the above solution, optionally, the write_new_file interface is compatible with the POSIX standard interface, and the operation of the interface simulates open, write and close functions through internal logic.

[0044] In the above solution, optionally, the method is applied to a distributed file system that supports high-concurrency data storage, and the distributed file system includes a metadata service, a data service, and a client.

[0045] Compared with the prior art, this application has at least the following beneficial effects:

[0046] Based on further analysis and research of existing technical problems, this application recognizes that the existing technology is difficult to meet the needs of efficient storage and access of massive small files. By designing a custom interface write_new_file and using data pipelines to store small file metadata and data, the efficiency problem of the existing distributed file system in small file processing is solved. Provide a custom file writing interface write_new_file, which directly receives the file path and complete data. Unlike the traditional POSIX interface, write_new_file does not need to be divided into three steps: open, write, and close, which simplifies the operation logic. Multiple data pipelines are created when the system is initialized. Each data pipeline is a linear storage space that can be mapped to multiple object storage devices (OSDs). These pipelines are used to store metadata and file contents of small files. Each time write_new_file is called, the system encapsulates the path, file data content, and metadata change record of the small file into a data packet. Each data packet contains a unique number, and the system uses the number as the basis for sorting when replaying the log. The data packet is written to the selected data pipeline, and the pipeline ID of the data packet and its position in the pipeline are recorded in the metadata. The client local cache records the pipeline position of the metadata. The background thread of the metadata service (MDS) replays the logs of the data pipeline according to the order of the data packet numbers, and writes the metadata changes and file contents to the traditional metadata storage area and file storage area respectively.

[0047] The write_new_file interface completes the file path parsing, metadata change and data storage operations through a single call, significantly reducing the multiple message interactions in traditional POSIX interface operations. The data and metadata of small files are packaged and written to the data pipeline at the same time, avoiding the multiple separate writes of metadata and file data in traditional systems, and reducing the total I / O times. The data pipeline is designed for multi-threaded concurrent writing, which can support multiple clients to operate simultaneously, greatly improving the throughput of the system in small file processing scenarios. The numbering of data packets makes log replay sequential and consistent, and can ensure that the metadata and file data in the system are fully synchronized after the write operation is completed. The present application effectively solves the problem of low I / O throughput in the scenario of new writing of massive small files mentioned in the background technology, and makes the system more efficient and reliable in practical applications by reducing interactions, reducing I / O and concurrent optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flowchart of a small file processing method based on a data pipeline provided in one embodiment of the present application;

[0049] Figure 2 A schematic diagram of the principle of a small file processing method based on a data pipeline provided in one embodiment of the present application;

[0050] Figure 3 A schematic diagram of the main process of write_new_file based on a data pipeline is provided for one embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] In one embodiment, Figure 2 As shown, a small file processing method based on a data pipeline is provided, comprising the following steps:

[0053] Provides a customized file writing interface write_new_file to receive the file path and complete data of a small file;

[0054] During the system startup initialization phase, multiple data pipelines are created to store metadata and data of small files. Each data pipeline is a linear space mapped to multiple object storage devices.

[0055] Through the write_new_file interface, the metadata changes and data content of the small file are packaged to generate a data packet containing the file path, data content and metadata change information;

[0056] Assign a unique number to each data packet and write it into the corresponding data pipeline;

[0057] Recording the pipeline ID of the data packet and its position in the pipeline in the metadata of the file;

[0058] In the background of the metadata service, the metadata and file data in the data pipeline are decomposed through log replay and stored in the predetermined storage space respectively.

[0059] This embodiment effectively solves the problem of low I / O throughput in the scenario of massive small file new writing mentioned in the background technology, and makes the system more efficient and reliable in practical applications by reducing interactions, lowering I / O and concurrency optimization.

[0060] In this embodiment, the data pipeline allocation method includes:

[0061] Allocation method based on polling mechanism;

[0062] Mapping allocation method based on file path hash value;

[0063] Dynamic allocation method based on real-time load status;

[0064] The number of data pipelines supports dynamic addition and deletion.

[0065] The polling and hash mapping in this embodiment ensure the uniform distribution of small files in multiple data pipelines, avoiding performance bottlenecks caused by overloading of certain pipelines. The dynamic allocation strategy evaluates pipeline load in real time, which can significantly improve pipeline utilization efficiency and avoid resource waste in high-concurrency scenarios. For small file paths with strong randomness or intensive writes, different allocation strategies can be flexibly adapted to ensure the stability and efficiency of the system in diverse scenarios.

[0066] In this embodiment, the write_new_file interface is executed by the following steps:

[0067] Send the file path and file data to the metadata service;

[0068] The metadata service performs a search operation on the file path. If the path already exists, the corresponding inode is reused. If the path does not exist, a new inode is created.

[0069] Encapsulate the path-related metadata change records and file data content into a data packet.

[0070] In this embodiment, one interface completes path resolution, metadata update, and data writing, without relying on multiple POSIX interface calls, reducing operational complexity. The simplified operation process reduces the risk of file writing failure due to operational errors or network problems, and improves system reliability. The interface design avoids the overhead of POSIX interface calls in traditional distributed file systems by integrating operations, and improves the writing speed of small files.

[0071] In this embodiment, the data pipeline supports multi-threaded concurrent writing, allowing multiple clients to access and write to multiple data pipelines at the same time.

[0072] The multi-threaded design of this embodiment fully utilizes the performance of multi-core CPUs and high-speed storage devices, and improves the throughput of data pipelines in high-concurrency scenarios. The thread pool and dynamic scheduling mechanism ensure the efficient use of system resources and avoid thread blocking or resource waste. The optimization of the lock mechanism ensures the integrity and consistency of multi-threaded concurrent writing, while avoiding the performance degradation caused by traditional large-granularity locks.

[0073] In this embodiment, the log replay process includes:

[0074] Read data packets sequentially from the data pipeline;

[0075] Extract metadata change records and file data contents according to the serial number sequence of the data packets;

[0076] Writing metadata change information into the metadata storage area;

[0077] Write the file data content to the data storage area.

[0078] The numbering and sorting mechanism of this embodiment ensures the order and consistency of metadata and file data during log replay, avoiding data confusion or loss. The metadata and data are processed separately, and the multi-threaded replay design greatly shortens the system recovery time. The state management mechanism allows the system to recover quickly when log replay fails, improving the reliability and stability of the system.

[0079] In this embodiment, the data pipeline is organized in the form of a log, where each data packet contains the following fields:

[0080] Package number;

[0081] File path information;

[0082] File data content;

[0083] File metadata change records;

[0084] Data verification information.

[0085] The data packet of this embodiment contains complete path information and data content, which reduces the search time during subsequent access. The data verification mechanism ensures the integrity of the data during writing and transmission, and can quickly recover even if an error occurs. The variable-size block design avoids space waste in fixed block allocation and improves storage utilization.

[0086] In this embodiment, before the log replay is completed, the file reading operation is implemented by the following steps:

[0087] The client queries the metadata service for the corresponding metadata based on the file path;

[0088] The data pipe ID and location recorded in the metadata are used to locate the target data packet;

[0089] The client reads the file data content directly from the data pipeline.

[0090] This embodiment obtains data by directly accessing the data pipeline before the log replay is completed, avoiding the traditional metadata search and data separation read and write operations. It supports normal access to data when the replay is not completed, improving the availability of the system. The local cache reduces the frequency of interaction between the client and the metadata service.

[0091] In this embodiment, after the log replay is completed, the file access operation is switched to the access mode of the traditional distributed file system, wherein the metadata and data are stored in the metadata storage area and the data storage area respectively.

[0092] In this embodiment, the write_new_file interface is compatible with the POSIX standard interface, and the operation of the interface simulates open, write and close functions through internal logic.

[0093] This embodiment provides a mechanism for dynamically adjusting the data pipeline load to optimize resource allocation and improve data processing efficiency. The specific implementation includes the following steps:

[0094] When the system is running, monitor the status of each data pipeline, including but not limited to the following indicators:

[0095] The current number of packets stored in the pipeline.

[0096] Write latency for the data pipeline.

[0097] The available storage capacity of the data pipeline.

[0098] This data is collected to generate a pipeline load report.

[0099] Based on the real-time load report, the following dynamic adjustment strategies are adopted:

[0100] When an increase in the overall system load is detected, new data pipelines are dynamically created.

[0101] When the pipeline load is reduced, excess pipeline resources are released through data migration and recycling.

[0102] Pipes with less load are given priority for accepting new data packets.

[0103] In extreme cases, it allows migrating existing packets from a high-load pipe to a low-load pipe.

[0104] The scheduler dynamically assigns packets to target pipes based on the following algorithm:

[0105] Select the pipeline with the lowest current load, combine the file path hash value with the pipeline's real-time load, and determine the target pipeline. Dynamically allocate data based on the remaining percentage of pipeline capacity. The system regularly evaluates the execution effect of the load balancing strategy and optimizes the next round of allocation strategy through a feedback mechanism.

[0106] This embodiment dynamically adjusts the number of pipelines and data distribution to make full use of the system's storage and computing resources and avoid resource waste. In high-concurrency write scenarios, load balancing reduces the risk of overloading a single pipeline, improving the stability and fault tolerance of the system. The intelligent scheduling algorithm reduces the response time of high-load pipelines and improves the efficiency of overall data processing. The ability to dynamically create and release pipeline resources allows the system to flexibly adapt to business needs of different scales.

[0107] In this embodiment, the method is applied to a distributed file system supporting high-concurrency data storage, and the distributed file system includes a metadata service, a data service, and a client.

[0108] In one embodiment, Figure 2 As shown, a small file performance optimization method based on data pipeline is provided, the method includes: customizing a private file writing interface write_new_file (interface parameters are file path and file data) to replace the original open / write / close interface function. For writing a new file, only one call to the write_new_file interface is required to pass the file path and all the data of the small file.

[0109] Create multiple data pipes to store metadata and data of small files. A data pipe is a linear space mapped to multiple OSDs.

[0110] The metadata changes and file data contents are packaged and numbered, and written into the corresponding data pipeline.

[0111] This embodiment improves the IO throughput of the scenario of writing a large number of small files, and the customized interface write_new_file is compatible with the existing open / read / write / close interfaces. In other words, the small files written by write_new_file can be accessed through the standard posix interface.

[0112] like Figure 3 As shown in the figure, the main process of write_new_file includes: the client sends a write new file message to the MDS, which carries the file path and data content.

[0113] The MDS performs a lookup operation to find the incoming path from the cache. If the inode already exists, it is reused. If not, it is queried from the OSD. If the file exists, its data is subsequently overwritten.

[0114] Create an inode memory structure and grant corresponding distributed lock permissions.

[0115] This write involves information changes of all inodes on the path. These metadata changes and file data are packaged and numbered.

[0116] Submit the packaged data to one of the data pipes. The data pipe can be selected in a round-robin manner, or in a hash mapping manner, or in other ways. The inode of the small file written this time needs to record the pipe ID and the location of the data in the pipe. At this point, the write_new_file interface returns.

[0117] MDS returns inode and distributed lock information to CLIENT for building the CLIENT local cache.

[0118] The background thread of MDS replays logs for all data pipelines, writing metadata and data to different spaces. Since the packaged data in the data pipeline is numbered, the data of multiple pipelines can ensure the order of log replay. Before the log replay is completed, MDS needs to retain the inode cache information and can directly access the data content of the file in the data pipeline.

[0119] The file written by the write_new_file interface in this embodiment can be accessed through the standard POSIX interface (open, read).

[0120] When the log is not replayed, the open interface will query the MDS for inode information. At this time, the inode cache hits, and the inode records the data location of the file (in a data pipeline). The Read interface obtains the file data from the data pipeline based on the data location information in the inode.

[0121] When log replay is complete, metadata and data are stored in the traditional way, and the inode clears the data pipeline location information. At this time, file access is consistent with the traditional method.

[0122] This solution improves the IO throughput in the scenario of writing a large number of small files. This is mainly reflected in the following aspects:

[0123] Reduces the number of message interactions.

[0124] Reduces the number of metadata and data IO.

[0125] The use of multiple data pipelines improves IO concurrency.

[0126] This embodiment uses a customized file writing interface write_new_file (interface parameters are file path and file data) to replace the original open / write / close interface functions, thereby reducing the number of message interactions for writing small files.

[0127] Introduce data pipelines to package metadata changes and data generated by small file write operations together and store them in the data pipeline.

[0128] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A small file processing method based on data pipeline, characterized in that: The method comprises: Provides a customized file writing interface write_new_file to receive the file path and complete data of a small file; During the system startup initialization phase, multiple data pipelines are created to store metadata and data of small files. Each data pipeline is a linear space mapped to multiple object storage devices. Through the write_new_file interface, the metadata changes and data content of the small file are packaged to generate a data packet containing the file path, data content and metadata change information; Assign a unique number to each data packet and write it into the corresponding data pipeline; Recording the pipeline ID of the data packet and its position in the pipeline in the metadata of the file; In the background of the metadata service, the metadata and file data in the data pipeline are decomposed through log replay and stored in the predetermined storage space respectively.

2. The method according to claim 1, characterized in that The data pipeline allocation method includes: Allocation method based on polling mechanism; Mapping allocation method based on file path hash value; Dynamic allocation method based on real-time load status; The number of data pipelines supports dynamic addition and deletion.

3. The method according to claim 1, characterized in that The write_new_file interface is executed by the following steps: Send the file path and file data to the metadata service; The metadata service performs a search operation on the file path. If the path already exists, the corresponding inode is reused. If the path does not exist, a new inode is created. Encapsulate the path-related metadata change records and file data content into a data packet.

4. The method according to claim 1, characterized in that The data pipeline supports multi-threaded concurrent writing, allowing multiple clients to access and write to multiple data pipelines at the same time.

5. The method according to claim 1, characterized in that The log replay process includes: Read data packets sequentially from the data pipeline; Extract metadata change records and file data contents according to the serial number sequence of the data packets; Writing metadata change information into the metadata storage area; Write the file data content to the data storage area.

6. The method according to claim 1, characterized in that The data pipeline is organized in log format, where each packet contains the following fields: Package number; File path information; File data content; File metadata change records; Data verification information.

7. The method according to claim 1, characterized in that Before log replay is completed, the file read operation is implemented through the following steps: The client queries the metadata service for the corresponding metadata based on the file path; The data pipe ID and location recorded in the metadata are used to locate the target data packet; The client reads the file data content directly from the data pipeline.

8. The method according to claim 1, characterized in that After the log replay is completed, the file access operation is switched to the access method of the traditional distributed file system, in which the metadata and data are stored in the metadata storage area and the data storage area respectively.

9. The method according to claim 1, characterized in that: The write_new_file interface is compatible with the POSIX standard interface, and the operation of the interface simulates open, write and close functions through internal logic.

10. The method according to claim 1, characterized in that The method is applied to a distributed file system supporting high-concurrency data storage, wherein the distributed file system includes a metadata service, a data service and a client.

Images