Progressive Data Striping Method for Task-Level Burst Buffer File System

Abstract

The present invention discloses a progressive data striping method for a task-level burst cache file system. The progressive data striping method for a task-level burst cache file system of the present invention includes configuring a striping component to perform progressive data striping on a file, initializing the file, and the file system operating on the striped file; the progressive data striping means that different parts of different file sizes are respectively segmented using different file block sizes and configured with different numbers of nodes for read and write operations, and starting from the head of the file, the size of each part monotonically increases, the file block size used monotonically increases, and the number of nodes used for read and write operations also monotonically increases. The purpose of the present invention is to improve the data storage and access performance of the task-level burst cache file system, and improve the data access efficiency and reliability in high-performance computing jobs.

Description

Technical Field

[0001] The present invention belongs to the technical field of data management of a burst cache file system, and particularly relates to a progressive data striping method for a task-level burst cache file system. Background Art

[0002] In high-performance computing (HPC) jobs, a large amount of data needs to be processed and analyzed, and storage also faces more complex patterns, including a huge number of metadata operations, small I / O requests, or random file I / Os, and high-speed data access is required to meet the needs of real-time computing and analysis. To improve data storage efficiency, burst cache technology has been widely used. A burst cache file system is usually a separate file system that can store temporary data for applications. It aggregates the high-speed local storage of computing nodes or uses a dedicated solid-state drive storage cluster to provide a peak higher than the backend bandwidth, reduce I / O operation latency, increase data throughput, and thus accelerate job applications.

[0003] However, with the increase in the data volume of HPC jobs and the complexity of computing tasks, traditional burst buffer file systems face the problem of being unable to efficiently process large-scale data and burst access loads. Existing cache management strategies usually adopt fixed cache strategies or static data striping techniques, which may lead to unbalanced utilization of cache resources and even cache overflow or performance bottlenecks under high loads. Therefore, how to improve the throughput of data access in high-performance computing jobs has become an urgent problem to be solved. To improve data throughput and optimize cache resource utilization, existing methods mainly include the following three: One is to use a fixed striping strategy. Its basic idea is to allocate data to multiple storage media according to a fixed strip size, so as to achieve parallel access. This method is suitable for systems with uniform loads and simple data access patterns and can improve throughput through parallel reading and writing. However, it is difficult to cope with dynamically changing loads and complex data access patterns. For scenarios with burst loads or frequently changing access patterns, fixed striping may cause some storage devices to be overloaded while other devices are idle, thus wasting storage resources. At the same time, the choice of strip size directly affects performance, but for different computing tasks and data access scenarios, choosing a fixed strip size is often not optimal. The second is to use cache prefetching and cache management strategies. Currently, some high-performance computing systems have adopted cache prefetching technology to reduce I / O latency by preloading likely-to-be-accessed data into the cache. Combined with intelligent cache management, data access can be predicted and scheduled according to the data access pattern, thereby improving the cache hit rate and data throughput. Due to relying on accurate prediction of the data access pattern, if the access pattern is unstable or difficult to predict, the prefetching strategy may have the opposite effect, resulting in cache pollution and resource waste, and it also has high requirements for the utilization of cache resources. If the cache resources are limited, the effect of the prefetching strategy may not be as expected. The third is to use a multi-level cache architecture by setting different levels of caches (for example, level-1 cache, level-2 cache, etc.) to more efficiently process data access requests at different levels. This method optimizes the data access path through a reasonable hierarchical structure and improves data throughput. However, such system architectures are complex and require managing multiple cache levels and data migration strategies. In addition, there are high requirements for tuning the cache levels and cache strategies. The cache sizes and usage strategies at different levels must be reasonably configured to achieve the best performance. In summary, existing methods can optimize the data access efficiency in high-performance computing jobs, but their respective limitations may also affect the overall efficiency and reliability of the system in certain specific scenarios. Summary of the Invention

[0004] Technical problems to be solved by the present invention: Aiming at the above problems of the prior art, a progressive data striping method for a task-level burst buffer file system is provided. The present invention aims to improve the data storage and access performance of the task-level burst buffer file system and enhance the data access efficiency and reliability in high-performance computing jobs.

[0005] To solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0006] A progressive data striping method for a task-level burst cache file system, comprising the following steps: configuring a striping component to perform progressive data striping on a file, initializing the file, and the file system operating on the striped file; the progressive data striping means that different parts of different file sizes are respectively segmented using different file block sizes and configured to use different numbers of nodes for read and write operations, and starting from the head of the file, the sizes of each part are monotonically increasing, the file block sizes used are monotonically increasing, and the number of nodes used for read and write operations is also monotonically increasing.

[0007] Optionally, when performing progressive data striping, it includes segmenting the part of the file from 0 to 4MB using a file block size of 512KB, and using 1 node to process the read and write operations; segmenting the part of the file from 4MB to 8MB using a file block size of 1MB, and using 3 nodes to process the read and write operations; segmenting the part of the file from 8MB to 16MB using a file block size of 2MB, and using 8 nodes to process the read and write operations; segmenting the part of the file from 16MB to 32MB using a file block size of 4MB, and using 16 nodes to process the read and write operations; segmenting the part of the file from 32MB to 64MB using a file block size of 8MB, and using 32 nodes to process the read and write operations; segmenting the part of the file after 64MB using a file block size of 16MB, and using all the remaining nodes to process the read and write operations.

[0008] Optionally, the initialization of the file includes calling a preset ID generation method generateID() in a specified configuration file config.hpp to initialize the file according to the configured progressive data striping configuration, segmenting it according to different file block sizes, and using an array to record the ID of each generated file block.

[0009] Optionally, the execution steps of the ID generation method generateID() include: initializing the file according to the configured progressive data striping configuration, segmenting it according to different file block sizes, calculating the file block start number chunkid of each file block of the striping component according to the start position and strip size of the striping component corresponding to each part of the file, and recording the file block number chunkid of each file block into the data PFLcomponents and returning it.

[0010] Optionally, the operations of the file system on the striped file include: forwarding the write request of the prepared file blocks to the file system server through the file system client, and parallel writing the corresponding file blocks into the file system by the file system server after receiving the request.

[0011] Optionally, the forwarding of the write request of the prepared file blocks to the file system server through the file system client includes: the file system client initializes the metadata for the newly created file, where the metadata includes the file name, file path, permissions, owner information, and striping configuration. The striping configuration includes parameters such as strip size and number of stripes, and assigns a unique identifier to the file; the file system client calls a hash function to determine the storage location of each prepared file block, calculates the overflow part of the given offset within the current file block size, checks whether the given offset is aligned to the boundary of the file block size by adjusting the offset and block size, calculates and returns the file block index corresponding to the given offset and block size, adjusts the offset and block size according to the configuration of the striping component, and uses this information to calculate the index; the file system client calls the forward_write() function to initiate a file block write request to the file system server, and after receiving the request, the file system server stores the file block in the file system; the execution of the forward_write() function includes:

[0012] S101, determine whether the size of the file is greater than 0. If not, generate an error prompt, end and exit; otherwise, jump to step S102;

[0013] S102, calculate the boundary and number of the file block start number chunkid of the current file block;

[0014] S103, determine whether the current file block is the last file block. If it is the last file block, jump to step S104; otherwise, jump to step S105;

[0015] S104, record the file block start number chunkid of the current file block, update the file block start numbers chunkid of the first and last file blocks, and expose the user buffer as the RDMA data source;

[0016] S105, send an RPC request to the file system server to make the file system server process the RPC request, and monitor whether all the file block data of the file is written to the file system server. If all the file block data of the file is written, end and exit; otherwise, generate an error prompt, end and exit.

[0017] Optionally, when the file system server writes the corresponding file block into the file system in parallel after receiving a request, it includes that the file system server first verifies the file block offset, and if the verification of the file block offset passes, it calls the RPC remote write function rpc_srv_write() to store the corresponding file block into the file system; the execution of the RPC remote write function rpc_srv_write() includes:

[0018] S201, set RPC information; allocate space for the bulk transfer buffer;

[0019] S202, process the file block start number chunkid of the file block passed in by the file system client through the RPC request;

[0020] S203, determine whether there are still file blocks to be processed. If there are still file blocks to be processed, jump to step S204; otherwise, jump to step S208;

[0021] S204, determine whether the file block is processed by the corresponding host host. If it is processed by the corresponding host host, jump to step S205; otherwise, jump to step S203;

[0022] S205, dynamically adjust the size of the current file block;

[0023] S206, determine whether the file block is the first or the last data block. If it is the first or the last data block, perform offset processing;

[0024] S207, transmit the file block and start an asynchronous write operation;

[0025] S208, wait for the task response, and send the result to the file system client after receiving the task response, and end and exit.

[0026] In addition, the present invention further provides a progressive data striping system for a task-level burst cache file system, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the progressive data striping method for the task-level burst cache file system.

[0027] In addition, the present invention further provides a computer-readable storage medium, in which a computer program or instruction is stored, and the computer program or instruction is programmed or configured to execute the progressive data striping method for the task-level burst cache file system through a processor.

[0028] In addition, the present invention also provides a computer program product, including a computer program or instructions, which are programmed or configured to execute the progressive data striping method for the task-level burst cache file system through a processor.

[0029] Compared with the prior art, the present invention can mainly achieve the following beneficial effects: The slice size is an important factor affecting I / O performance because it directly affects parallelism, cache hit rate, and the management overhead of the file system. Reasonably selecting the slice size can balance performance and resource usage, and improve the throughput and response speed of the system. An overly large slice may lead to resource waste, while an overly small slice may result in increased management and transmission overheads. By optimizing the slice size according to different application scenarios and data access patterns, the I / O performance of the system can be significantly improved. Aiming at the problems existing in the prior art, the progressive data striping method for the task-level burst cache file system of the present invention performs striping processing on files in a differentiated manner, adopts different slice sizes for different parts of the file, and configures different numbers of storage nodes at the same time, reducing unnecessary I / O requests and delays, improving data storage performance and access efficiency, and further enhancing the throughput and response speed of the system. The present invention uses a progressive data striping strategy to perform progressive data striping and splitting on files, without splitting files using the same block size, and additionally allocates different numbers of nodes to process different striping components. In application scenarios such as large-scale concurrent access, large file storage, and high-performance computing, it can effectively reduce the I / O load of the system and improve the parallel read and write performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the basic process of the method in the embodiment of the present invention.

[0031] Figure 2 It is a schematic diagram of progressive data striping in the embodiment of the present invention.

[0032] Figure 3 It is a schematic diagram of a striping component in the embodiment of the present invention.

[0033] Figure 4 It is a schematic diagram of the basic process of the forward_write() function for forward writing in the embodiment of the present invention.

[0034] Figure 5 It is a schematic diagram of the basic process of the rpc_srv_write() function for RPC remote writing in the embodiment of the present invention.

[0035] Figure 6 It is a schematic diagram of the performance comparison of writing in the embodiment of the present invention.

[0036] Figure 7Schematic diagram of performance comparison for reading in the embodiments of the present invention. Detailed implementation manners

[0037] To enable those skilled in the art of the present technology to better understand the technical solutions of the present invention, the technical solutions of the present invention will be further described in detail below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0038] As Figure 1 shown, the progressive data striping method for the task-level burst cache file system in this embodiment includes the following steps: configuring a striping component to perform progressive data striping on a file, initializing the file, and the file system operating on the striped file; the progressive data striping means that each part with a different file size is separately segmented using different file block sizes and configured with different numbers of nodes for read and write operations, and starting from the head of the file, the size of each part increases monotonically, the file block size used increases monotonically, and the number of nodes used for read and write operations also increases monotonically. In high-performance computing scenarios, a large number of file read and write operations are involved. When processing file read and write operations, the slice size of the file is an important factor affecting data storage and access performance, as well as system I / O performance. When the slice size is too small, the concurrency performance can be improved, but each time a smaller slice is accessed, the system must perform additional metadata processing. Especially in a distributed file system, each slice requires a separate I / O request and metadata lookup. Therefore, if the slice is too small, the frequency of I / O requests will increase significantly. In addition, small slices mean that more blocks need to be accessed, which may lead to more frequent access to different storage devices or storage blocks during each read or write, which may increase the addressing overhead of the storage device and reduce the overall performance. If the slice size is increased too much, the number of I / O requests can be reduced, but each large slice may occupy the resources of multiple storage nodes, resulting in overloading of some nodes while other nodes are idle, unable to fully utilize the parallel processing ability of the cluster. In addition, if the slice is too large while the actual amount of data to be accessed is small, the bandwidth of the storage system will be wasted. Therefore, in order to optimize the read and write performance of files in high-concurrency situations, the method in this embodiment uses a differential method to perform progressive data striping on files, adopts different slice sizes for different parts of the file, and configures different numbers of storage nodes at the same time, reducing unnecessary I / O requests and delays, improving data storage performance and access efficiency, and further improving the throughput and response speed of the system.

[0039] As Figure 2As shown in the figure, when progressive data striping is performed in this embodiment, it includes splitting the 0 to 4MB part of the file using a file block size of 512KB, and the read and write operations are processed using 1 node; splitting the 4MB to 8MB part of the file using a file block size of 1MB, and the read and write operations are processed using 3 nodes; splitting the 8MB to 16MB part of the file using a file block size of 2MB, and the read and write operations are processed using 8 nodes; splitting the 16MB to 32MB part of the file using a file block size of 4MB, and the read and write operations are processed using 16 nodes; splitting the 32MB to 64MB part of the file using a file block size of 8MB, and the read and write operations are processed using 32 nodes; splitting the part of the file after 64MB using a file block size of 16MB, and the read and write operations are processed using all the remaining nodes (62 in this embodiment), thus forming a progressive layout of the file (Progressive File Layout, PFL). As Figure 3 As shown, there are six striping components in this embodiment: the first striping component contains the first 4MB of the file and only 1 node is used to process the file, the second striping component contains the range from 4MB to 8MB of the file and 3 nodes are used, the third striping component contains the range from 8MB to 16MB of the file and 8 nodes are used, until the last striping component uses all the nodes.

[0040] In this embodiment, initializing the file includes calling the preset ID generation method generateID() in the specified configuration file config.hpp to initialize the file according to the configured progressive data striping configuration, splitting it according to different file block sizes, and using an array to record the ID of each generated file block.

[0041] In this embodiment, the execution steps of the ID generation method generateID() include: initializing the file according to the configured progressive data striping configuration, splitting it according to different file block sizes, calculating the file block start number chunkid of each file block of the striping component according to the start position and strip size of the striping component corresponding to each part of the file, and recording the file block number chunkid of each file block into the data PFLcomponents and returning it. Specifically in this embodiment, then the return value is passed to the parameter variable PFLchunkID, and the parameter variable PFLchunkID records the file block start ID (i.e., the file block start number chunkid) of each striping component.

[0042] In this embodiment, the operations performed by the file system on the striped files include: forwarding the write request of the prepared file blocks to the file system server through the file system client, and the file system server writing the corresponding file blocks into the file system in parallel after receiving the request.

[0043] In this embodiment, forwarding the write request of the prepared file blocks to the file system server through the file system client includes: the file system client initializes the metadata for the newly created file, and the metadata includes the file name, file path, permissions, owner information, and striping configuration, where the striping configuration includes, for example, the stripe size and the number of stripes, and assigns a unique identifier to the file; the file system client calls a hash function to determine the storage location of each prepared file block, calculates the overflow part of the given offset within the current file block size, checks whether the given offset is aligned to the boundary of the file block size by adjusting the offset and the block size, calculates and returns the file block index corresponding to the given offset and block size, adjusts the offset and the block size according to the configuration of the striping component, and uses this information to calculate the index; the file system client calls the forward_write() function to initiate a file block write request to the file system server, and the file system server stores the file block in the file system after receiving the request; as Figure 4 shown, the execution of the forward_write() function in this embodiment includes:

[0044] S101, determine whether the size of the file is greater than 0. If not, generate an error prompt, end and exit; otherwise, jump to step S102;

[0045] S102, calculate the boundary and number of the file block start number chunkid of the current file block;

[0046] S103, determine whether the current file block is the last file block. If it is the last file block, jump to step S104; otherwise, jump to step S105;

[0047] S104, record the file block start number chunkid of the current file block, update the file block start numbers chunkid of the first and last file blocks, and expose the user buffer as the RDMA data source;

[0048] S105, send an RPC request to the file system server to make the file system server process the RPC request, and monitor whether all the file block data of the file is written to the file system server. If all the file block data of the file is written, end and exit; otherwise, generate an error prompt, end and exit.

[0049] In this embodiment, when the file system server writes the corresponding file blocks into the file system in parallel after receiving a request, it includes that the file system server first verifies the file block offset. If the verification of the file block offset passes, it calls the RPC remote write function rpc_srv_write() to store the corresponding file block into the file system; as Figure 5 shown, the execution of the RPC remote write function rpc_srv_write() in this embodiment includes:

[0050] S201, set RPC (Remote Procedure Call, used for communication between the client and the storage server) information. Configuring the RPC information means setting the parameters related to the client and the storage server; allocate space for the bulk transfer buffer;

[0051] S202, process the file block start number chunkid of the file block passed in by the file system client through the RPC request;

[0052] S203, determine whether there are still file blocks to be processed. If there are still file blocks to be processed, jump to step S204; otherwise, jump to step S208;

[0053] S204, determine whether the file block is processed by the corresponding host host. If it is processed by the corresponding host host, jump to step S205; otherwise, jump to step S203. In this embodiment, by determining whether the file block is processed by the corresponding host host, the temporary file system used realizes the decentralized distribution of file blocks to different hosts and allows them to process the corresponding data blocks respectively, reducing the single-point bottleneck and improving the I / O performance;

[0054] S205, dynamically adjust the size of the current file block. Oversized slices may cause resource waste, and undersized slices may cause increased management and transmission overhead. By optimizing the slice size according to different application scenarios and data access patterns, the I / O performance of the system can be significantly improved, that is, adjusted according to the previous Progressive File Layout (PFL);

[0055] S206, determine whether the file block is the first or the last data block. If it is the first or the last data block, perform offset processing; by judging whether the number of the current data block is equal to the start block number or the end block number, to determine whether it is the first or the last data block. If it is the first data block, the excess part in the file offset needs to be subtracted; if it is the last data block, the part less than a complete file block at the end of the file needs to be subtracted to ensure that the data can be correctly aligned and stored;

[0056] S207, transmit the file block and start the asynchronous write operation;

[0057] S208, Wait for the task response, and after receiving the task response, send the result to the file system client, then end and exit.

[0058] The file system client calls the forward_write() method to perform the file writing operation, forwarding the prepared file block write request to the file system server. After receiving the request, the file system server calls the rpc_srv_write() method to write the corresponding file blocks into the file system in parallel. First, the file system initializes the metadata for the newly created file. The metadata includes the file name, file path, permissions, owner information, and striping configuration (such as stripe size and number of stripes), and assigns a unique identifier to the file. Then, it calls the forward_write() method to initiate a request to the file system server. After receiving the request, the file system server calls the rpc_srv_write() method to store the file blocks requested by the client in the file system.

[0059] To verify the progressive data striping method for the task-level burst buffer file system in this embodiment, IOR is used for testing in this embodiment. IOR (Input / Output Review) is a widely used I / O performance testing tool, usually used to evaluate the throughput and performance of the storage subsystem in a high-performance computing system. By simulating high-load I / O scenarios of parallel file systems (such as Lustre, GPFS, BeeGFS, etc.), it tests and analyzes the performance of the storage system. To evaluate the read and write performance of progressive data striping in a high-performance computing environment, the IOR test tool is used to conduct read and write performance tests and compare with traditional file systems using non-progressive data striping. The test file sizes are gradually increased from 128MB, 256MB, 512MB, 1024MB to 2048MB, and the average value is taken after running each test 10 times. The test results are shown in Figure 6 and Figure 7 , Figure 6 shows the comparative experimental results of the IOR test read operation, Figure 7 shows the comparative experimental results of the IOR test write operation, and the unit of the test results is MB / S. See Figure 6 and Figure 7It can be seen that progressive data striping performs better in both writing and reading performance. Especially when dealing with large data blocks (such as 2048MB), the performance improvement is most obvious. This indicates that the progressive data striping method can make more effective use of storage resources and optimize performance by dynamically adjusting the stripe size. In the case of different block sizes, the performance growth of progressive data striping is relatively large, while non - progressive data striping shows relatively stable and inefficient performance. Especially for small block sizes, it fails to effectively improve the bandwidth utilization rate. Generally speaking, progressive data striping can provide higher throughput and lower latency under different read - write loads, and is suitable for application scenarios that need to process a large amount of data and high concurrency. Non - progressive data striping is more suitable for scenarios with uniform loads, but in the case of complex tasks or large amounts of data, the performance improvement is limited.

[0060] In summary, the progressive data striping adopted in the progressive data striping method for the task - level burst cache file system in this embodiment is a method that optimizes the file system performance by gradually adjusting the file striping strategy according to the file layout. In a high - concurrency scenario, the striping of files directly affects the way data is stored and accessed. A smaller striping unit will significantly increase the number of I / O requests, while a larger striping unit, although reducing the number of I / O requests, may lead to greater latency because each I / O request is larger and takes longer to complete. In addition, a larger striping may result in excessive data reading / writing, causing waste of disk bandwidth. Therefore, using the progressive data striping strategy to allocate different - sized stripes to different parts of the file, this progressive adjustment strategy can appropriately split the file, thereby improving I / O efficiency, reducing latency, and further enhancing the file read - write performance in a high - concurrency environment.

[0061] In addition, this embodiment also provides a progressive data striping system for a task - level burst cache file system, including a microprocessor and a memory connected to each other. The microprocessor is programmed or configured to execute the progressive data striping method for the task - level burst cache file system.

[0062] In addition, this embodiment also provides a computer - readable storage medium. The computer - readable storage medium stores a computer program or instruction, and the computer program or instruction is programmed or configured to execute the progressive data striping method for the task - level burst cache file system through a processor.

[0063] In addition, this embodiment also provides a computer program product, including a computer program or instruction, and the computer program or instruction is programmed or configured to execute the progressive data striping method for the task - level burst cache file system through a processor.

[0064] Those skilled in the art should understand that the technical solutions provided by the present invention can be in the form of a method, a system, or a computer program product. Therefore, the present invention can be implemented in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can be in the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) that contain computer-usable program code. The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for realizing the functions specified in Figure 1 one or more processes and / or blocks Figure 1 one or more blocks. These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device realizes the functions specified in Figure 1 one or more processes and / or blocks Figure 1 one or more blocks. These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Therefore, the instructions executed on the computer or other programmable device provide steps for realizing the functions specified in Figure 1 one or more processes and / or blocks Figure 1 one or more blocks.

[0065] The above is only the preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions within the idea of the present invention belong to the protection scope of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements should also be regarded as within the protection scope of the present invention.

Claims

1. A progressive data striping method for a task-level burst cache file system, characterized in that It includes the following steps: configuring a striping component to perform progressive data striping on a file, initializing the file, and the file system operating on the striped file; the progressive data striping means that different parts of the file with different sizes are respectively sliced using different file block sizes and configured to use different numbers of nodes for read and write operations, and starting from the head of the file, the sizes of each part increase monotonically, the file block sizes used increase monotonically, and the number of nodes used for read and write operations also increases monotonically; When performing progressive data striping, it includes slicing the part of the file from 0 to 4MB using a file block size of 512KB, and processing the read and write operations using 1 node; Slicing the part of the file from 4MB to 8MB using a file block size of 1MB, and processing the read and write operations using 3 nodes; slicing the part of the file from 8MB to 16MB using a file block size of 2MB, and processing the read and write operations using 8 nodes; slicing the part of the file from 16MB to 32MB using a file block size of 4MB, and processing the read and write operations using 16 nodes; slicing the part of the file from 32MB to 64MB using a file block size of 8MB, and processing the read and write operations using 32 nodes; slicing the part of the file after 64MB using a file block size of 16MB, and processing the read and write operations using all the remaining nodes.

2. The progressive data striping method for a task-level burst cache file system according to claim 1, wherein The initialization of the file includes calling a preset ID generation method generateID() in the specified configuration file config.hpp to initialize the file according to the configured progressive data striping configuration, slicing according to different file block sizes, and using an array to record the IDs of each generated file block.

3. The progressive data striping method for a task-level burst cache file system according to claim 2, wherein The execution steps of the ID generation method generateID() include: initializing the file according to the configured progressive data striping configuration, slicing according to different file block sizes, calculating the file block start number chunkid of each file block of the striping component according to the start position and strip size of the striping component corresponding to each part of the file, and recording the file block numbers chunkid of each file block into the data PFLcomponents and returning.

4. The progressive data striping method for the task-level burst cache file system according to claim 1, wherein The operation of the file system on the striped file includes: forwarding the prepared file block write request to the file system server through the file system client, and the file system server writing the corresponding file block into the file system in parallel after receiving the request.

5. The progressive data striping method for a task-level burst cache file system according to claim 4, wherein The operation of forwarding the prepared file block write request to the file system server through the file system client includes: The file system client initializes the metadata for the newly created file. The metadata includes the file name, file path, permissions, owner information, and striping configuration. The striping configuration includes parameters such as stripe size and number of stripes, and assigns a unique identifier to the file. The file system client calls a hash function to determine the storage location of each prepared file block, calculates the overflow part of the given offset within the current file block size, checks whether the given offset is aligned to the boundary of the file block size by adjusting the offset and block size, calculates and returns the file block index corresponding to the given offset and block size, adjusts the offset and block size according to the configuration of the striping component, and uses this information to calculate the index. The file system client calls the forward_write() function to initiate a file block write request to the file system server. After receiving the request, the file system server stores the file block in the file system. The execution of the forward_write() function includes: S101, Determine whether the size of the file is greater than 0. If not, generate an error prompt, end and exit; otherwise, jump to step S102; S102, Calculate the boundary and number of the file block starting number chunkid of the current file block; S103, Determine whether the current file block is the last file block. If it is the last file block, jump to step S104; otherwise, jump to step S105; S104, Record the file block starting number chunkid of the current file block, update the file block starting numbers chunkid of the first and last file blocks, and expose the user buffer as the RDMA data source; S105, Send an RPC request to the file system server to enable the file system server to process the RPC request, and monitor whether all file block data of the file is written to the file system server. If all file block data of the file is written, end and exit; otherwise, generate an error prompt, end and exit.

6. The progressive data striping method for a task-level burst cache file system according to claim 4, wherein When the file system server parallel writes the corresponding file block to the file system after receiving the request, it includes that the file system server first verifies the file block offset. If the verification of the file block offset passes, it calls the rpc_srv_write() RPC remote write function to store the corresponding file block in the file system. The execution of the rpc_srv_write() RPC remote write function includes: S201, Set the RPC information; allocate space for the bulk transfer buffer; S202, Process the file block starting number chunkid of the file block passed in by the file system client through the RPC request; S203, Determine whether there are still file blocks to be processed. If there are still file blocks to be processed, jump to step S204; otherwise, jump to step S208; S204, Determine whether the file block is processed by the corresponding host host. If it is processed by the corresponding host host, jump to step S205; otherwise, jump to step S203; S205, Dynamically adjust the size of the current file block; S206, Determine whether the file block is the first or the last data block. If it is the first or the last data block, perform offset processing; S207, Transmit the file block and start an asynchronous write operation; S208, Wait for the task response, and send the result to the file system client after receiving the task response, and end and exit.

7. A progressive data striping system for a task-level burst cache file system, comprising a microprocessor and a memory connected to each other, characterized in that, The microprocessor is programmed or configured to execute the progressive data striping method for the task-level burst cache file system according to any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program or instructions, characterized in that, The computer program or instruction is programmed or configured to execute the progressive data striping method for the task-level burst cache file system according to any one of claims 1 to 6 through a processor.

9. A computer program product, comprising a computer program or instructions, characterized in that, The computer program or instruction is programmed or configured to execute the progressive data striping method for the task-level burst cache file system according to any one of claims 1 to 6 through a processor.

Images