Method for effectively improving read-write efficiency and concurrency of small file use storage

Abstract

The invention discloses a method for effectively improving read-write efficiency and concurrency of small file use storage, which comprises the following steps of: merging a plurality of small files in storage, and packaging the plurality of small files into a storage block; adjusting a read-write scheduling algorithm according to the access mode of the file to ensure that the frequently accessed file is preferentially loaded; and a lock mechanism and read-write priority control are introduced to realize concurrency control, so that data conflicts cannot be generated during concurrent access of multiple threads or processes to small files, and meanwhile, the concurrency capability is maximized. According to the method, storage fragments are reduced by combining the small files, the scheduling algorithm is optimized, the access response speed is increased, concurrency is controlled, data conflicts are avoided, the read-write efficiency and concurrent processing capacity of small file storage are effectively improved, and the method is suitable for small file intensive storage scenes.

Description

Technical Field

[0001] This invention relates to the field of storage technology, and more specifically to a method for effectively improving the read and write efficiency and concurrency of small file storage. Background Technology

[0002] Traditional storage systems were not optimized for scenarios with a large number of small files from the outset, resulting in significant performance bottlenecks when handling such files. On one hand, low read / write efficiency stems from the mismatch between storage medium characteristics and the random access patterns of small files, leading to multiple independent storage accesses for a single file operation and amplifying I / O overhead. On the other hand, poor concurrency management arises from the centralized metadata architecture's tendency to create access hotspots, making it difficult to support high-concurrency metadata query and operation demands, thus impacting overall system responsiveness. These problems become increasingly pronounced as data volumes grow, hindering efficiency improvements in related application scenarios. Summary of the Invention

[0003] To address the shortcomings of existing technologies, the present invention aims to provide a method for effectively improving the read and write efficiency and concurrency of small file storage. By using file merging, scheduling algorithm optimization, and concurrency control mechanisms, the performance bottleneck problem of traditional storage systems when processing a large number of small files is solved.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a method for effectively improving the read / write efficiency and concurrency of small file storage, comprising the following steps: Step 1: Merge multiple small files in storage and package them into a single storage block; Step 2: Adjust the read / write scheduling algorithm according to the file access pattern to ensure that frequently accessed files are loaded first. Step 3: Introduce locking mechanisms and read / write priority control to achieve concurrency control, ensuring that concurrent access to small files by multiple threads or processes will not cause data conflicts, while maximizing concurrency capabilities.

[0005] As a further improvement to the present invention, the specific method for merging multiple small files in step one is as follows: Set the size of each file in the storage system to Where i is the file index, and the size of each file satisfies: in, This is the maximum capacity of the storage block. The files to be merged are then inserted into the storage block to complete the file merging process.

[0006] As a further improvement of the present invention, in the process of merging multiple small files in step one, a merging factor M is defined, representing the number of files to be merged in a storage block, satisfying: This merge factor M needs to be dynamically adjusted to adapt to changes in file size, thereby ensuring that as many small files as possible are merged without exceeding the storage block capacity.

[0007] As a further improvement to the present invention, the specific method for adjusting the read / write scheduling algorithm in step two is as follows: setting the priority function of the access mode. and , representing the priorities of read and write operations respectively, and the priority function is calculated based on the file access history and access frequency: in, and These are the file read / write rates, calculated as the ratio of the file access frequency to the past read or write time.

[0008] As a further improvement of the present invention, in the concurrency control of step three, a thread concurrency level is set. This represents the maximum number of concurrent threads the system can handle simultaneously. The concurrency control strategy uses locking mechanisms to ensure the atomicity of each read / write operation and defines a concurrency scheduling function. Its value depends on the current number of threads in the system and the size of the task queue: in, For the file The corresponding length of the waiting access queue.

[0009] The beneficial effects of this invention are as follows: by merging multiple small files into a storage block, the number of I / O operations in the storage system is reduced, significantly improving read and write efficiency; the scheduling algorithm is adjusted according to the access pattern, prioritizing frequently accessed files and optimizing resource allocation; and a locking mechanism and priority control are introduced to achieve concurrency control, ensuring data consistency for multi-threaded access while maximizing concurrency capabilities. This solution effectively solves the performance bottleneck of traditional storage systems through multi-dimensional collaborative optimization. Improvements such as dynamically adjusting the merging factor, calculating the priority function based on access frequency, and setting thread concurrency levels respectively enhance the adaptability of file merging, the accuracy of the scheduling algorithm, and the flexibility of concurrency control. Detailed Implementation

[0010] The present invention will now be described in further detail with reference to the embodiments.

[0011] This embodiment provides a method for effectively improving the read / write efficiency and concurrency of small files, comprising the following steps: Step 1: Merge multiple small files in storage and package them into a single storage block; Step 2: Adjust the read / write scheduling algorithm according to the file access pattern to ensure that frequently accessed files are loaded first. Step three involves introducing a locking mechanism and read / write priority control to achieve concurrency control, ensuring that concurrent access to small files by multiple threads or processes does not cause data conflicts, while maximizing concurrency capabilities. This method reduces IO overhead through file merging, improves resource utilization through scheduling algorithm optimization, and resolves access conflicts through a concurrency control mechanism, thereby effectively improving the read / write efficiency and concurrency capabilities of small file storage, achieving the beneficial effects of this invention.

[0012] Furthermore, the specific method for merging multiple small files in step one is as follows: Set the size of each file in the storage system to Where i is the file index, and the size of each file satisfies: in, This is the maximum capacity of the storage block. The files to be merged are then placed into the storage block to complete the file merging. This method, by reasonably setting the storage block capacity, ensures that small files are merged without exceeding the capacity limit, thus helping to improve read and write efficiency.

[0013] Furthermore, in the process of merging multiple small files in step one, a merge factor M is defined, representing the number of files merged in a storage block, satisfying: This merge factor M needs to be dynamically adjusted to adapt to changes in file size, thereby ensuring that as many small files as possible are merged without exceeding the storage block capacity. This method improves the adaptability of file merging and further optimizes read and write efficiency.

[0014] Furthermore, the specific method for adjusting the read / write scheduling algorithm in step two is as follows: Set the priority function of the access mode. and , representing the priorities of read and write operations respectively, and the priority function is calculated based on the file access history and access frequency: in, and These are the file read and write rates, calculated as the ratio of the file access frequency to the past read or write time. This method improves the accuracy of the scheduling algorithm and optimizes resource allocation.

[0015] Furthermore, in the concurrency control of step three, a thread concurrency level is set. This represents the maximum number of concurrent threads the system can handle simultaneously. The concurrency control strategy uses locking mechanisms to ensure the atomicity of each read / write operation and defines a concurrency scheduling function. Its value depends on the current number of threads in the system and the size of the task queue: in, For the file The corresponding length of the waiting access queue. This method improves the flexibility of concurrency control, maximizing concurrency capabilities while ensuring data consistency.

[0016] In summary, this invention employs a combination of file merging, scheduling algorithm optimization, and concurrency control to solve the problems of low read / write efficiency and poor concurrent access management in traditional storage systems when processing a large number of small files. It achieves a significant improvement in the read / write efficiency of small file storage and an effective enhancement of concurrency capabilities, demonstrating high practical value and technological advancement.

[0017] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for effectively improving the read / write efficiency and concurrency of small file storage, characterized in that: Includes the following steps: Step 1: Merge multiple small files in storage and package them into a single storage block; Step 2: Adjust the read / write scheduling algorithm according to the file access pattern to ensure that frequently accessed files are loaded first. Step 3: Introduce locking mechanisms and read / write priority control to achieve concurrency control, ensuring that concurrent access to small files by multiple threads or processes will not cause data conflicts, while maximizing concurrency capabilities.

2. The method for effectively improving the read / write efficiency and concurrency of small file storage according to claim 1, characterized in that: The specific method for merging multiple small files in step one is as follows: Set the size of each file in the storage system to Where i is the file index, and the size of each file satisfies: in, This is the maximum capacity of the storage block. The files to be merged are then inserted into the storage block to complete the file merging process.

3. The method for effectively improving the read / write efficiency and concurrency of small file storage according to claim 2, characterized in that: In the process of merging multiple small files in step one, a merge factor M is defined, representing the number of files merged in a storage block, satisfying the following: This merge factor M needs to be dynamically adjusted to adapt to changes in file size, thereby ensuring that as many small files as possible are merged without exceeding the storage block capacity.

4. The method for effectively improving the read / write efficiency and concurrency of small file storage according to any one of claims 1 to 3, characterized in that: The specific method for adjusting the read / write scheduling algorithm in step two is as follows: Set the priority function for the access mode. and , representing the priorities of read and write operations respectively, and the priority function is calculated based on the file access history and access frequency: in, and These are the file read / write rates, calculated as the ratio of the file access frequency to the past read or write time.

5. The method for effectively improving the read / write efficiency and concurrency of small file storage according to any one of claims 1 to 3, characterized in that: In the concurrency control of step three, a thread concurrency level is set. This represents the maximum number of concurrent threads the system can handle simultaneously. The concurrency control strategy uses locking mechanisms to ensure the atomicity of each read / write operation and defines a concurrency scheduling function. Its value depends on the current number of threads in the system and the size of the task queue: in, For the file The corresponding length of the waiting access queue.

Images