A small file storage method, device, equipment, medium and program product

Abstract

The application provides a small file storage method, device, equipment, medium and program product. The method comprises the following steps: performing hash calculation on the storage path of a small file, determining a target large file bucket to which the small file is to be stored based on the obtained hash value, and the target large file bucket corresponding to a target metadata bucket and a target file name bucket; when the sum of the data amount of the small file and the data amount of the target large file bucket exceeds a preset bucket capacity threshold, generating a new large file bucket, a new metadata bucket and a new file name bucket under the same directory of the target large file bucket, and migrating the stored part of the small file, metadata and file name in the target large file bucket, the target metadata bucket and the target file name bucket to the new large file bucket, the new metadata bucket and the new file name bucket, so as to store the small file, the metadata thereof and the file name of the small file into the target large file bucket, the target metadata bucket and the target file name bucket respectively. The application can significantly improve the storage performance and reading efficiency of the small file.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage, and in particular to a small file storage method, device, equipment, medium and program product. BACKGROUND

[0002] In recent years, with the rapid development of mobile applications such as social media, short video and e-commerce, the number of small files generated has increased dramatically. These small files are of various types, different sizes and frequently updated, and their number can reach tens of millions, hundreds of millions or even billions, which makes their storage a big problem in the industry. Distributed file systems have been widely used in data storage due to their high reliability, easy scalability and high fault tolerance. However, they mainly optimize the read-write performance of large files and lack effective optimization means for small file read-write operations. This leads to a significant increase in resource consumption and time consumption of small file operations such as addition, deletion, search and modification in the case of massive small files, which greatly reduces the performance of distributed file systems in handling massive small files and makes it difficult to meet the actual needs of commercial users.

[0003] To solve this problem, industry technical personnel mostly optimize from the hardware and software levels.

[0004] At the hardware level, high-performance read-write of small files is mainly achieved by combining high-speed solid-state disks. However, this method requires expensive hardware devices when handling massive small files, resulting in a significant increase in storage costs. In addition, hardware devices are easily affected by external environmental factors, causing data loss and posing a major risk to data security.

[0005] At the software level, the read-write performance of small files is mainly optimized by aggregating small files. However, this method is only suitable for the storage of a small number of small files, and when the number of small files increases to a massive level, the read-write performance of small files decreases dramatically. The reason is that this method simply combines multiple small files into one large file without optimizing the read-write process of small files, so that when massive small files are read, the large aggregated file still needs to be traversed one by one, making it impossible to quickly locate the target file, resulting in a significant increase in file read-write pressure and a serious loss of read-write performance. SUMMARY

[0006] To overcome the problems in the related art, the present application provides a small file storage method, device, equipment, medium and program product.

[0007] According to a first aspect of an embodiment of the present application, a small file storage method is provided, the method comprising:

[0008] hashing a storage path of the small file, determining, based on a hash value obtained, a target large file bucket to which the small file is to be stored, the target large file bucket being used to aggregate and store a plurality of small files, each of the target large file buckets corresponding to a target metadata bucket and a target filename bucket, the target metadata bucket and the target filename bucket being used to store metadata and filenames of small files in the corresponding target large file bucket, respectively;

[0009] when a sum of a data amount of the small file and a data amount of the target large file bucket exceeds a preset bucket capacity threshold, generating a new large file bucket, a new metadata bucket and a new filename bucket under a same directory as the target large file bucket, and migrating part of the small files, metadata and filenames stored in the target large file bucket, the target metadata bucket and the target filename bucket to the new large file bucket, the new metadata bucket and the new filename bucket, so as to store the small files, the metadata of the small files and the filenames of the small files to the target large file bucket, the target metadata bucket and the target filename bucket, respectively;

[0010] wherein the target metadata bucket stores the metadata of the small file by using a red-black tree data structure, and storing the metadata of the small file to the target metadata bucket specifically includes: performing a lexicographic comparison between the filename of the small file and filenames of red-black tree nodes in the target metadata bucket, and determining a storage position of the metadata of the small file in the target metadata bucket based on a comparison result;

[0011] after the storage of the small file is completed, if a retrieval request for the small file is received, determining, based on a hash value corresponding to the small file, a target large file bucket and a target metadata bucket to which the small file is stored, retrieving the metadata of the small file in the target metadata bucket according to the filename of the small file, and extracting the small file from the target large file bucket according to the retrieved metadata.

[0012] According to a second aspect of the embodiments of the present application, a small file storage device is provided, and the device includes:

[0013] a large file bucket determination module configured to hash a storage path of the small file, determine, based on a hash value obtained, a target large file bucket to which the small file is to be stored, the target large file bucket being used to aggregate and store a plurality of small files, each of the target large file buckets corresponding to a target metadata bucket and a target filename bucket, the target metadata bucket and the target filename bucket being used to store metadata and filenames of small files in the corresponding target large file bucket, respectively;

[0014] The small file storage module is configured to, when the sum of the data amount of the small file and the data amount of the target large file bucket exceeds a preset bucket capacity threshold, generate a new large file bucket, a new metadata bucket and a new file name bucket under a same level directory of the target large file bucket, and migrate the stored part of the small file, metadata and file name in the target large file bucket, target metadata bucket and target file name bucket to the new large file bucket, new metadata bucket and new file name bucket, so as to store the small file, metadata of the small file and file name of the small file into the target large file bucket, target metadata bucket and target file name bucket respectively.

[0015] The target metadata bucket stores the metadata of the small file in a red-black tree data structure, and the metadata of the small file is stored in the target metadata bucket by performing a lexicographical comparison between the file name of the small file and the file name of a red-black tree node in the target metadata bucket, and determining the storage position of the metadata of the small file in the target metadata bucket based on the comparison result.

[0016] After the storage of the small file is completed, if a retrieval request for the small file is received, the target large file bucket and the target metadata bucket to which the small file is aggregated are determined according to the hash value corresponding to the small file, the metadata of the small file is retrieved in the target metadata bucket according to the file name of the small file, and the small file is extracted from the target large file bucket according to the retrieved metadata.

[0017] According to a third aspect of the embodiments of the present application, a computer device is provided, including a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of the first aspect when executing the program.

[0018] According to a fourth aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores a computer program, and the program is executed by a processor to implement the method of the first aspect.

[0019] According to a fifth aspect of the embodiments of the present application, a computer program product is provided, including a computer program, and the computer program is executed by a processor to implement the method of the first aspect.

[0020] The technical solutions provided by the embodiments of the present application can have the following beneficial effects:

[0021] In the embodiments of the present application, the mapping relationship between the hash value of the small file and the index of the large file bucket is constructed by performing hash calculation on the storage path of the small file, so that when the small file is retrieved, the large file bucket where the small file is stored can be quickly located according to the hash value, the resource consumption caused by traversing the large files one by one in the distributed file system is avoided, the reading burden of the small file is reduced, and the reading efficiency of the small file is significantly improved. At the same time, the storage condition in the large file bucket can also be monitored in real time in the embodiments of the present application, and when the data amount in the large file bucket approaches the bucket capacity threshold, the bucket splitting and file migration are dynamically performed, so that the read-write request pressure of a single large file bucket is effectively dispersed, the storage structure is timely adjusted, the single point overload is avoided, the concurrent processing capability of the distributed file system is significantly improved, and the stability and reliability of the storage performance are guaranteed.

[0022] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0023] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application.

[0024] Figure 1 FIG. 1 is a flowchart of a small file storage method according to an exemplary embodiment of the present application.

[0025] Figure 2 FIG. 2 is a schematic diagram of the storage condition of the small file in the large file bucket according to an exemplary embodiment of the present application.

[0026] Figure 3 FIG. 3 is a schematic diagram of file migration according to an exemplary embodiment of the present application.

[0027] Figure 4 FIG. 4 is a flowchart of another small file storage method according to an exemplary embodiment of the present application.

[0028] Figure 5 FIG. 5 is a hardware structure diagram of a computer device where a small file storage device is located according to an exemplary embodiment of the present application.

[0029] Figure 6 FIG. 6 is a structure block diagram of a small file storage device according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0030] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The following description is made with reference to the accompanying drawings in which like reference numerals refer to like elements, unless the context of use indicates otherwise. The following description of exemplary embodiments is not representative of all embodiments consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0031] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0032] It is to be understood that the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0033] Before delving into the details of the present application, it is necessary to briefly explain some basic concepts involved in the present application, so as to better understand the content of the present application.

[0034] Small file: refers to a file with a small amount of data. Considering the storage performance and aggregation effect, the industry generally refers to a file with a data amount less than 8 KB as a small file.

[0035] Large file bucket: can be understood as a special file or container for aggregating and managing small files under the same level storage directory. Each large file bucket can correspond to a metadata bucket and a file name bucket, respectively, for storing the metadata and file name of the small files in the corresponding large file bucket. The metadata of a file refers to data that can describe the file attribute characteristics and storage state, usually including the following information of the file: file name, file size, creation time, modification time, permission information, etc. In actual operation, the size of the large file bucket and its metadata bucket and file name bucket can be 32 MB or 64 MB.

[0036] Red-black tree: a self-balancing binary search tree with the following important properties: each node is either red or black; the root node is black; leaf nodes (NIL nodes, empty nodes) are black; each red node has two black children (there cannot be two consecutive red nodes on any path from each leaf to the root); all paths from any node to its leaves contain the same number of black nodes. These properties ensure the self-balancing of the red-black tree, so that in the insertion, deletion and lookup operations, the height of the tree is kept at logarithmic level, thereby improving the operation efficiency.

[0037] ASCII code: an encoding scheme based on English characters, used to represent characters in text data. ASCII code uses 7-bit binary numbers to represent 128 different characters, including uppercase and lowercase English letters, digits 0-9, and some control characters and punctuation marks. In computer systems, ASCII code is widely used for the storage and transmission of text files, and is a universal character encoding standard.

[0038] In order to solve the problems existing in the related art, the present application provides a new small file storage method. The method constructs the index mapping relationship between the hash value of the small file and the large file bucket by performing hash calculation on the storage path of the small file, so that when searching for the small file, the large file bucket where the small file is stored can be quickly located according to the hash value, avoiding the resource consumption caused by traversing the large file one by one in the distributed file system, reducing the reading burden of the small file, and significantly improving the reading efficiency of the small file. At the same time, the method can also monitor the storage situation in the large file bucket in real time, and when the data amount in the large file bucket approaches the bucket capacity threshold, dynamically perform bucket splitting and file migration, so as to effectively disperse the read-write request pressure of a single large file bucket, timely adjust the storage structure, avoid single point overload, significantly improve the concurrent processing capability of the distributed file system, and guarantee the stability and reliability of the storage performance.

[0039] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0040] Figure 1 is a flowchart of a small file storage method according to an exemplary embodiment of the present application. As shown in Figure 1 , the method comprises the following steps:

[0041] S101, performing hash calculation on the storage path of the small file, determining the target large file bucket where the small file is to be stored based on the obtained hash value, the target large file bucket being used to aggregate and store a plurality of small files, each target large file bucket corresponding to a target metadata bucket and a target filename bucket, the target metadata bucket and the target filename bucket being used to store the metadata and the filename of the small file in the corresponding target large file bucket, respectively;

[0042] S102, when the sum of the data amount of the small files and the data amount of the target large file bucket exceeds the preset bucket capacity threshold, generating a new large file bucket, a new metadata bucket and a new filename bucket under the same directory as the target large file bucket, and migrating the stored part of the small files, metadata and filenames in the target large file bucket, the target metadata bucket and the target filename bucket to the new large file bucket, the new metadata bucket and the new filename bucket, so as to store the small files, the metadata of the small files and the filenames of the small files into the target large file bucket, the target metadata bucket and the target filename bucket respectively.

[0043] In step S101, since the distributed file system consumes certain resources for storing each file, when facing mass small file storage, the resource consumption is huge, which leads to resource waste and performance bottleneck. Therefore, a large file bucket can be created under the storage path of the small file, and multiple small files under the same storage directory are aggregated into a large file to form a large file bucket for storage, so as to save resource consumption, reduce the read-write times of the distributed file system for small files, and improve the storage performance of small files. In addition, the storage path of the small file can be calculated by using a hash algorithm to obtain a hash value corresponding to the small file uniquely, and the hash value is used to determine the large file bucket to which the small file is stored, so as to construct the index mapping relationship between the hash value and the large file bucket. When searching for the small file subsequently, the hash value can be used to quickly locate the large file bucket where the small file is stored, so as to avoid the resource consumption caused by traversing the large file in the distributed file system one by one, reduce the read burden of the small file, and significantly improve the read efficiency of the small file. The hash algorithm used for hash calculation can be selected according to actual user needs, which is not limited in the present application.

[0044] Specifically, in some embodiments, in the process of determining the target large file bucket, non-character symbols in the storage path, such as " / ", "\", ":" and the like, can be removed first. These symbols have no actual meaning in determining the large file bucket, and removing them can simplify the subsequent calculation process and reduce the computational complexity. Next, the processed storage path is converted into corresponding ASCII code values. This conversion process converts character information into numerical form, facilitating subsequent numerical processing and calculation. However, the length of the converted ASCII code value data is still relatively long and may contain a lot of unnecessary information in determining the large file bucket. Therefore, in order to reduce the influence of these unnecessary information, the ASCII code value can also be subjected to data dimension reduction processing. Dimension reduction processing can be achieved through various methods, such as principal component analysis (PCA) or linear discriminant analysis (LDA), etc. These methods can extract the main features of the data and remove redundant information, thereby improving the efficiency and accuracy of data processing. After dimension reduction processing, the data volume is reduced, making the hash calculation more efficient and the obtained hash value more able to reflect the essential characteristics of the small file storage path. Subsequently, the dimension-reduced ASCII code value is subjected to hash calculation to obtain the hash value corresponding to the small file, and the hash value is subjected to modulo processing to obtain the bucket sequence number of the target large file bucket where the small file is to be stored, based on which the target large file bucket is determined. This process can map the storage path of the small file to a value within a limited range, so that this range corresponds to the number of large file buckets, and the obtained value is the bucket sequence number of the target large file bucket where the small file is to be stored. In this way, each hash value can correspond to a unique bucket sequence number, and thus determine the specific large file bucket where the small file should be stored, achieving the function of quickly locating the large file bucket according to the hash value. In addition, the hash value has good dispersion, which can uniformly distribute different small files into different buckets, avoiding the situation that some buckets are overloaded while others are idle, thereby achieving load balancing and ensuring the efficient operation of the distributed file system.

[0045] In step 102, after determining the target large file bucket to which the small file is to be stored, the sum of the data amount of the small file and the data amount of the target large file bucket can be further compared with the preset bucket capacity threshold to determine whether the target large file bucket has enough space to accommodate the current small file to be stored. If the sum does not exceed the threshold, it means that the target large file bucket still has enough space, and the current small file can be normally stored in the target large file bucket, and the metadata and the file name of the small file can also be stored in the corresponding target metadata bucket and target file name bucket. If the sum exceeds the threshold, it means that the target large file bucket cannot accommodate the current small file to be stored, and at this time, a new large file bucket and its corresponding new metadata bucket and new file name bucket can be generated under the same directory as the target large file bucket, and the data stored in the target large file bucket, the target metadata bucket and the target file name bucket can be migrated to the corresponding new buckets to release the space of the target large file bucket, the target metadata bucket and the target file name bucket, so that the current small file to be stored can be stored. In this way, the storage situation in the large file bucket can be monitored in real time, and when the data amount in the large file bucket approaches the bucket capacity threshold, the bucket splitting and file migration can be dynamically performed, so as to effectively disperse the read-write request pressure of a single large file bucket, timely adjust the storage structure, avoid single-point overload, significantly improve the concurrent processing capability of the distributed file system, and guarantee the stability and reliability of the storage performance.

[0046] In the process of storing the small file into the target large file bucket, if the small files are not stored in the large file bucket in the order of data amount, some scattered and discontinuous spaces, i.e., file fragments, can be generated after some small files are deleted. These file fragments can cause the subsequent storage of other files to require additional space when the file size does not match the size of the fragment space, thereby causing waste of the originally generated file fragment resources. Therefore, in order to avoid the waste of file fragment resources and optimize the utilization rate of the storage space in the large file bucket, in some embodiments, the small files in the target large file bucket can be stored in order of the data amount of the small files. The data amount order can be from small to large or from large to small, and the specific order can be set according to the actual needs of the user, which is not limited in the present application.

[0047] With a specific example, the above case is described: suppose the data size of the small files to be stored in a large file bucket is 30, 20, 10, 5, 5, 2, 1, 1, 1, 1 in turn. In the case of storing in order, the stored small files can be 30, 20, 10, 5, 5, 2, 1, 1, 1, 1 or 1, 1, 1, 1, 5, 5, 10, 20, 30; and in the case of not storing in order, the stored small files can be 1, 10, 1, 5, 30, 1, 2, 20, 5, 1. If the small files with data size of 30, 20, 10 are deleted subsequently, in the case of storing in order, a continuous space of 60 size will be formed; and in the case of not storing in order, scattered spaces of 30, 20, 10 will be left. At this time, if 3 files with size of 15 need to be stored, in the case of storing in order, no additional space needs to be applied, while in the case of not storing in order, additional space needs to be applied.

[0048] In the implementation process, when a small file is stored in a target large file bucket, the bucket size of the target large file bucket before the small file is stored can be taken as a starting point, and the small file can be stored in order according to the data size, and the size of the storage is the data size of the small file. In addition, a custom end symbol such as "\0" can be used to end the small file to separate the small files in the target large file bucket. Figure 2 FIG. 1 is a schematic diagram of storage of a small file in a large file bucket according to an example embodiment of the present application. After each small file is stored, the offset of the small file in the large file bucket can be updated to the metadata of the small file, so as to facilitate subsequent quick positioning and acquisition of the target small file in the target large file bucket according to the offset.

[0049] In some embodiments, the metadata of the small file can specifically include the file name of the small file, the offset of the small file in the large file bucket, the data size of the small file, the creation and modification time of the small file, and the storage root node information of the small file. These metadata information can comprehensively describe the attribute characteristics and storage state of the small file, facilitating efficient management and retrieval of the small file. The file name can uniquely identify the small file, facilitating the use of the file name to realize the indexing of the metadata of the small file; the offset and the data size are helpful for quick positioning and reading of the small file in the target large file bucket; the creation and modification time can provide a reference for the time attribute of the small file; and the storage root node information can indicate the node storage location of the small file in the distributed file system.

[0050] In order to realize efficient metadata management and retrieval function, the target metadata bucket can store the metadata of the small file by using a red-black tree data structure. As a self-balancing binary search tree, the red-black tree can ensure that the height of the tree is maintained at a logarithmic level, thereby maintaining a low time complexity in data insertion, deletion and retrieval operations, and the red-black tree occupies a small memory overhead, so it can be well applied to the small file storage application scenarios of efficient retrieval and dynamic update.

[0051] When storing the metadata of the small file into the target metadata bucket, the file name of the small file can be compared with the file name of the red-black tree node in the target metadata bucket in lexicographical order, and the storage position of the metadata of the small file in the target metadata bucket is determined based on the comparison result. Specifically, if the lexicographical order of the file name of the small file is greater than the lexicographical order of the file name of the current red-black tree node, the right subtree of the current red-black tree node is recursively searched until there is no red-black tree node whose file name is smaller than the file name of the small file, and the position of the search end is determined as the storage position of the metadata of the small file in the target metadata bucket. Conversely, if the lexicographical order of the file name of the small file is smaller than the lexicographical order of the file name of the current red-black tree node, the left subtree of the current red-black tree node is recursively searched until there is no red-black tree node whose file name is greater than the file name of the small file, and the position of the search end is determined as the storage position of the metadata of the small file in the target metadata bucket. This storage strategy based on the red-black tree can effectively guarantee the order of the small file metadata and optimize the retrieval efficiency of the metadata. Even in the scenario of a large number of small files, the metadata of the file can still be quickly and accurately located and managed.

[0052] For the file name in the target file name bucket, some embodiments can be stored in order from early to late according to the creation time of the small file. This storage method facilitates the ordered migration of files according to the creation time when file migration is required, thereby improving the efficiency and manageability of the migration process. In the specific implementation process, each file name in the file name bucket can also be separated by a horizontal tab character such as “\t” to ensure clear distinction and easy processing of the file name.

[0053] When it is determined that the stored parts of the small files, the metadata and the filenames in the target large file bucket, the target metadata bucket and the target filename bucket need to be migrated to the new large file bucket, the new metadata bucket and the new filename bucket, the filenames in the target filename bucket with the creation time in the later half part can be migrated to the new filename bucket according to the creation time first; then, the metadata to be migrated is determined in the target metadata bucket according to the migrated filenames, and the metadata to be migrated is migrated to the new metadata bucket; finally, the small files to be migrated are determined in the target large file bucket according to the metadata to be migrated, and the small files to be migrated are migrated to the new large file bucket. The storage mode of the small files and the metadata and the filenames thereof to be migrated in the new large file bucket, the new metadata bucket and the new filename bucket is similar to that in the original target large file bucket, the target metadata bucket and the target filename bucket, and specific details can be referred to the foregoing description, which will not be described here.

[0054] Through the above steps, the file migration operation is completed, and the data in the original target large file bucket, the target metadata bucket and the target filename bucket and the data in the newly generated new large file bucket, the new metadata bucket and the new filename bucket are evenly distributed, so as to release the space in the original target large file bucket, the target metadata bucket and the target filename bucket, so that the current small files and the metadata and the filenames thereof to be stored can be continuously stored, and the situation of single bucket overload is effectively avoided. Figure 3 is a file migration schematic diagram according to an example embodiment of the present application.

[0055] On this basis, since the migrated files are stored in the new large file bucket, the bucket serial number will change, and the bucket serial number is calculated by the hash value of the small file, so the hash value corresponding to the small file also needs to be updated accordingly, otherwise, when the small file is retrieved, the original large file bucket may be incorrectly located according to the original hash value, resulting in failure to retrieve the corresponding file. That is, after the small files to be migrated are migrated to the new large file bucket, the hash value corresponding to the small files to be migrated also needs to be updated according to the bucket serial number of the new large file bucket, so as to ensure the correct indexing and retrieval of the files.

[0056] In combination with the above embodiments, in order to enable a clearer understanding of the present application, the storage process of the small file storage method provided by the present application is exemplarily described below through a specific application example.

[0057] Figure 4 is a flowchart of another small file storage method according to an example embodiment of the present application. As shown in Figure 4 , the method comprises the following steps:

[0058] S401, initialize the distributed file system. This step can specifically include: setting the mounting and storage path, ensuring that the file system can perform file read and write operations according to the path consistent with the user's needs; registering the file system information in the kernel, so that the operating system can identify and manage the file system; connecting the VFS (virtual file system) interface, managing the file system through the super block, which can contain the control information of the file system, such as the size of the file system and the block size; initializing the inode root node information, which is used to describe the directory attributes of the file system and indicates the starting point of all files and directories in the file system.

[0059] S402, determine whether the currently stored file is a small file. If yes, continue to execute step S403, if not, execute the normal large file storage mechanism of the distributed file system. In this step, whether the currently stored file is a small file can be determined by comparing whether the data amount of the currently stored file meets the preset data amount range of the small file. For example, if a file with a data amount less than 8k is a small file, it can be determined whether the data amount of the currently stored file is less than 8k. If yes, it is determined to be a small file, if not, it is determined to be a large file.

[0060] S403, perform hash calculation on the storage path of the small file, and determine the target large file bucket to which the small file is to be stored based on the obtained hash value.

[0061] S404, determine whether the sum of the data amount of the small file and the data amount of the target large file bucket exceeds the preset bucket capacity threshold. If yes, continue to execute step S405, if not, directly execute step S407.

[0062] S405, generate a new large file bucket, a new metadata bucket and a new file name bucket under the same directory of the target large file bucket;

[0063] S406, migrate the file names in the target file name bucket whose creation time is in the latter half to the new file name bucket, determine the metadata to be migrated in the target metadata bucket according to the migrated file names, migrate the metadata to be migrated to the new metadata bucket, and then determine the small files to be migrated in the target large file bucket according to the metadata to be migrated, migrate the small files to be migrated to the new large file bucket, and after storing the small files to be migrated, update the offset of the small files to be migrated in the new large file bucket to the metadata of the small files.

[0064] S407, store the small file, its metadata and file name to the target large file bucket, the target metadata bucket and the target file name bucket respectively, and after storing the small file, update the offset of the small file in the target large file bucket to the metadata of the small file.

[0065] S408, updating the hash value corresponding to the small file to be migrated according to the bucket sequence number of the new large file bucket.

[0066] After the storage of the small file is completed based on the above-mentioned embodiments, if a retrieval request for the small file is received, the target large file bucket and the target metadata bucket to which the small file is stored can be determined according to the hash value corresponding to the small file, the metadata of the small file is retrieved in the target metadata bucket according to the file name of the small file, and the small file is extracted from the target large file bucket according to the retrieved metadata.

[0067] Specifically, the bucket sequence number of the target large file bucket to which the small file is stored can be obtained by performing a remainder operation on the hash value of the small file, so as to determine the target large file bucket and the target metadata bucket. Then, in the target metadata bucket, the file name of the small file is compared with the file name in the red-black tree node in a lexicographical order to determine the storage position of the metadata of the small file in the target metadata bucket, so as to obtain the metadata of the small file. Finally, the storage position of the small file in the target large file bucket is determined according to the offset in the metadata, so as to extract the corresponding small file from the target large file bucket. This way quickly locates the large file bucket through the hash value, and realizes efficient retrieval and fast access of the small file.

[0068] Corresponding to the above-mentioned embodiments of the small file storage method, the present application also provides an embodiment of a small file storage device and a terminal to which the small file storage device is applied.

[0069] The embodiment of the small file storage device of the present application can be applied to a computer device, such as a server or a terminal device. The device embodiment can be realized by software, or by hardware or a combination of software and hardware. Taking software realization as an example, as a logical device, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory and running by the processor. From the hardware level, as shown in Figure 5 , it is a hardware structure diagram of a computer device where the small file storage device of the embodiment of the present application is located. In addition to the processor 501, the memory 502, the network interface 503, and the non-volatile memory 504 shown in Figure 5 , the server or electronic device where the small file storage device in the embodiment is located usually includes other hardware according to the actual function of the computer device, which will not be described here.

[0070] Figure 6 is a structural block diagram of a small file storage device according to an exemplary embodiment of the present application. As shown in Figure 6 , the device includes:

[0071] The large file bucket determination module 601 is configured to perform hash calculation on the storage path of the small file, determine a target large file bucket to which the small file is to be stored based on the obtained hash value, and the target large file bucket is configured to aggregate storage of a plurality of small files, each target large file bucket corresponds to a target metadata bucket and a target filename bucket, and the target metadata bucket and the target filename bucket are respectively configured to store metadata and filenames of the small files in the corresponding target large file bucket;

[0072] The small file storage module 602 is configured to, when the sum of the data amount of the small file and the data amount of the target large file bucket exceeds a preset bucket capacity threshold, generate a new large file bucket, a new metadata bucket and a new filename bucket under the same directory as the target large file bucket, and migrate the stored part of the small files, metadata and filenames in the target large file bucket, the target metadata bucket and the target filename bucket to the new large file bucket, the new metadata bucket and the new filename bucket, so as to store the small files, the metadata of the small files and the filenames of the small files into the target large file bucket, the target metadata bucket and the target filename bucket respectively.

[0073] The target metadata bucket stores the metadata of the small file by using a red-black tree data structure, and the metadata of the small file is stored into the target metadata bucket, specifically including: performing a lexicographic comparison between the filename of the small file and the filename of the red-black tree node in the target metadata bucket, and determining the storage position of the metadata of the small file in the target metadata bucket based on the comparison result.

[0074] After the storage of the small file is completed, if a retrieval request for the small file is received, the target large file bucket and the target metadata bucket to which the small file is stored are determined according to the hash value corresponding to the small file, the metadata of the small file is retrieved in the target metadata bucket according to the filename of the small file, and the small file is extracted from the target large file bucket according to the retrieved metadata.

[0075] Correspondingly, the present application also provides a computer device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the small file storage method described in any of the above embodiments when executing the computer program.

[0076] Correspondingly, the present application also provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the small file storage method described in any of the above embodiments when executed by a processor.

[0077] Correspondingly, the present application also provides a computer program product, which comprises a computer program, and the computer program implements the steps of the small file storage method described in any of the above embodiments when executed by a processor.

[0078] The implementation process of the functions and roles of each module in the above device is specifically described in the implementation process of the corresponding steps in the above method, which will not be repeated here.

[0079] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts are referred to the part of the method embodiments. The device embodiments described above are only illustrative, wherein the modules described as separate components can or can not be physically separated, and the components shown as modules can or can not be physical modules, i.e., can be located in one place or distributed on multiple network modules. Part or all of the modules can be selected to achieve the purposes of the present application according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0080] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims can be performed in a different order and still achieve desirable results. Also, the processes depicted in the figures do not necessarily require the particular order shown or sequential order in order to achieve the desired results. In some implementations, multitasking and parallel processing can be advantageous.

[0081] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the application be limited only by the scope of the claims, including any amendments thereof, and can include equivalents. The specification and examples given are intended as illustrative only and not limiting of the true scope and spirit of the application.

[0082] It should be understood that the application is not limited to the precise construction and compositions described above and shown in the attached drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application is limited only by the claims that follow, and the full scope of equivalents thereof.

[0083] The above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of protection of the present application.

Claims

1. A small file storage method characterized by comprising: The application comprises the following steps: Hash calculation is performed on the storage path of a small file, and a target large file bucket to which the small file is to be stored is determined based on the obtained hash value, the target large file bucket being used for aggregative storage of multiple small files, each target large file bucket corresponding to a target metadata bucket and a target file name bucket, the target metadata bucket and the target file name bucket being used for storage of metadata and file names of small files in the corresponding target large file bucket respectively; When the sum of the data amount of the small file and the data amount of the target large file bucket exceeds a preset bucket capacity threshold, a new large file bucket, a new metadata bucket and a new file name bucket are generated under the same directory of the target large file bucket, and part of the small files, metadata and file names stored in the target large file bucket, the target metadata bucket and the target file name bucket are correspondingly migrated to the new large file bucket, the new metadata bucket and the new file name bucket, so as to store the small files, the metadata of the small files and the file names of the small files into the target large file bucket, the target metadata bucket and the target file name bucket respectively; The target metadata bucket adopts a red-black tree data structure to store the metadata of the small file, and the metadata of the small file is stored into the target metadata bucket, specifically including: performing a lexicographical comparison between the file name of the small file and the file name of a red-black tree node in the target metadata bucket, and determining the storage position of the metadata of the small file in the target metadata bucket based on the comparison result; After the storage of the small file is completed, if a retrieval request for the small file is received, the target large file bucket and the target metadata bucket to which the small file is stored are determined according to the hash value corresponding to the small file, the metadata of the small file is retrieved in the target metadata bucket according to the file name of the small file, and the small file is extracted from the target large file bucket according to the retrieved metadata.

2. The method of claim 1, wherein, The application comprises the following steps: Non-character symbols in the storage path of the small file are removed, and the processed storage path is converted into corresponding ASCII code values; Dimension reduction is performed on the ASCII code values, and hash calculation is performed on the dimension-reduced ASCII code values to obtain a hash value corresponding to the small file; The hash value is subjected to a modulo operation to obtain a bucket serial number of the target large file bucket to which the small file is to be stored, and the target large file bucket is determined based on the bucket serial number.

3. The method of claim 1, wherein, The small files in the target large file bucket are sequentially stored in order of the data amount of the small files.

4. The method of claim 1, wherein, Based on the comparison result, the storage position of the metadata of the small file in the target metadata bucket is determined, including: If the lexicographical order of the file name of the small file is greater than the lexicographical order of the file name of the current red-black tree node, a right subtree of the current red-black tree node is recursively searched until there is no red-black tree node whose file name has a lexicographical order smaller than that of the file name of the small file, and the position of the search end is determined as the storage position of the metadata of the small file in the target metadata bucket; If the lexicographical order of the file name of the small file is less than the lexicographical order of the file name of the current red-black tree node, recursively search the left subtree of the current red-black tree node until there is no red-black tree node whose file name has a lexicographical order greater than the lexicographical order of the file name of the small file, and determine the position where the search ends as the storage position of the metadata of the small file in the target metadata bucket.

5. The method of claim 1, wherein, The file names in the target file name bucket are stored in order of the creation time of the small files from early to late; Corresponding parts of the small files, metadata and file names stored in the target large file bucket, target metadata bucket and target file name bucket are migrated to the new large file bucket, new metadata bucket and new file name bucket, including: According to the creation time, the file names in the target file name bucket whose creation time is in the later half are migrated to the new file name bucket; According to the migrated file names, determine the metadata to be migrated in the target metadata bucket, and migrate the metadata to be migrated to the new metadata bucket; According to the metadata to be migrated, determine the small files to be migrated in the target large file bucket, and migrate the small files to be migrated to the new large file bucket.

6. The method of claim 5, wherein, After migrating the small files to be migrated to the new large file bucket, further including: According to the bucket serial number of the new large file bucket, update the hash value corresponding to the small file to be migrated.

7. A small file storage apparatus characterized by comprising: Including: A large file bucket determination module is configured to perform hash calculation on the storage path of a small file, determine a target large file bucket to which the small file is to be stored based on the obtained hash value, and aggregate multiple small files in the target large file bucket. Each target large file bucket corresponds to a target metadata bucket and a target file name bucket, and the target metadata bucket and the target file name bucket are respectively used to store the metadata and the file name of the small file in the corresponding target large file bucket. A small file storage module is configured to generate a new large file bucket, a new metadata bucket and a new file name bucket under the same directory as the target large file bucket when the sum of the data amount of the small file and the data amount of the target large file bucket exceeds a preset bucket capacity threshold, and migrate corresponding parts of the small files, metadata and file names stored in the target large file bucket, target metadata bucket and target file name bucket to the new large file bucket, new metadata bucket and new file name bucket, so as to store the small file, the metadata of the small file and the file name of the small file in the target large file bucket, target metadata bucket and target file name bucket respectively. The target metadata bucket stores the metadata of the small file in a red-black tree data structure; and the metadata of the small file is stored in the target metadata bucket, specifically including: comparing the file name of the small file with the file name of the red-black tree node in the target metadata bucket in lexicographical order, and determining the storage position of the metadata of the small file in the target metadata bucket based on the comparison result. After the storage of the small file is completed, if a retrieval request for the small file is received, a target large file bucket and a target metadata bucket to which the small file is stored are determined according to a hash value corresponding to the small file, metadata of the small file is retrieved in the target metadata bucket according to a file name of the small file, and the small file is extracted from the target large file bucket according to the retrieved metadata.

8. A computer device, comprising: A computer program product comprising a memory, a processor and a computer program stored on the memory and loadable into the processor, wherein the processor implements the method of any one of claims 1 to 6 when executing the program.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1 to 6. The computer program is executed by the processor to implement the method of any one of claims 1 to 6.

Images