Methods, apparatus and computer program products for storage management

Abstract

Embodiments of this disclosure provide a method, apparatus, and computer program product for storage management. The method for storage management includes obtaining a read request for a first target file, the read request including a first target filename of the first target file. The method further includes determining a first target characteristic value for the first target file based on the first target filename. The method further includes determining first target index information for the first target file based on the first target characteristic value from a mapping between characteristic values ​​and index information, the first target index information being used to index the first target file from a merged file. The method further includes reading the first target file included in the merged file from a storage system based on the first target index information. Thus, access to a large number of small files can be transformed into access to large files, improving overall file access efficiency.

Description

Technical Field

[0001] Embodiments of this disclosure generally relate to storage systems, and more particularly to a method, electronic device, and computer program product for storage management. Background Technology

[0002] Files are stored in storage systems. To read a stored file, many storage systems require reading the file's directory entry and inode to determine its location within the system, and then retrieving the actual file content based on that location. Currently, with the development of various applications, a large number of small files are generated daily. Under current file access methods, accessing small files leads to excessive input / output (I / O) access, resulting in poor performance. Summary of the Invention

[0003] Embodiments of this disclosure relate to an improvement for storage management.

[0004] In a first aspect of this disclosure, a method for storage management is provided. The method includes obtaining a read request for a first target file, the read request including a first target filename of the first target file; determining a first target characteristic value for the first target file based on the first target filename; determining first target index information for the first target file from a mapping between the characteristic value and index information based on the first target characteristic value, the first target index information being used to index the first target file from a merged file, the merged file including multiple sections for storing multiple files and stored in a storage system, the multiple files including the first target file; and reading the first target file included in the merged file from the storage system based on the first target index information.

[0005] In a second aspect of this disclosure, an electronic device is provided. The electronic device includes at least one processor; and at least one memory storing computer program instructions, the at least one memory and the computer program instructions being configured to, together with the at least one processor, cause the electronic device to perform actions. The actions include: acquiring a read request for a first target file, the read request including a first target filename of the first target file; determining a first target characteristic value for the first target file based on the first target filename; determining first target index information for the first target file from a mapping between the characteristic value and index information based on the first target characteristic value, the first target index information being used to index the first target file from a merged file, the merged file including multiple sections for storing multiple files and stored in a storage system, the multiple files including the first target file; and reading the first target file included in the merged file from the storage system based on the first target index information.

[0006] In a third aspect of this disclosure, a computer program product is provided. The computer program product is tangibly stored on a non-volatile computer-readable medium and includes computer-executable instructions. When executed, the computer-executable instructions cause a device to perform the method of the first aspect.

[0007] It should be understood that the description in the Summary of the Invention section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0008] The above and other objects, features, and advantages of embodiments of the present disclosure will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of the present disclosure are illustrated in the drawings by way of example and not limitation, in which:

[0009] Figure 1 A schematic diagram of a typical storage environment is shown;

[0010] Figure 2 A schematic diagram of a storage environment according to some embodiments of the present disclosure is shown;

[0011] Figure 3 Examples of merged documents according to some embodiments of this disclosure are shown;

[0012] Figure 4 Examples of mapping between characteristic values ​​and index information according to some embodiments of this disclosure are shown;

[0013] Figure 5 A block diagram of a storage environment according to other embodiments of the present disclosure is shown;

[0014] Figure 6 Examples of index files according to some embodiments of this disclosure are shown;

[0015] Figure 7 A flowchart illustrating a file reading process according to some embodiments of the present disclosure is shown;

[0016] Figure 8 A flowchart illustrating a file writing process according to some embodiments of the present disclosure is shown;

[0017] Figure 9 A flowchart illustrating a file deletion process according to some embodiments of the present disclosure is shown;

[0018] Figure 10 A flowchart illustrating a storage space reclamation process according to some embodiments of the present disclosure is shown; and

[0019] Figure 11A schematic block diagram of an apparatus that can be used to implement embodiments of the present disclosure is shown.

[0020] Throughout all the accompanying drawings, the same or similar reference numerals are used to denote the same or similar components. Detailed Implementation

[0021] The principles and spirit of this disclosure will now be described with reference to several exemplary embodiments illustrated in the accompanying drawings. It should be understood that these specific embodiments are described only to enable those skilled in the art to better understand and implement this disclosure, and are not intended to limit the scope of this disclosure in any way.

[0022] The term “comprising” and its variations, as used herein, signify open inclusion, meaning “including but not limited to”. Unless otherwise stated, the term “or” means “and / or”. The term “based on” means “at least partially based on”. The terms “one example implementation” and “one implementation” mean “at least one example implementation”. The term “another implementation” means “at least one additional implementation”. The terms “first,” “second,” etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0023] Figure 1 A schematic diagram of this typical storage environment is shown. Figure 1 In the example environment 100, storage management system 110 is configured to control and manage storage system 120, including storing data in storage system 120, providing access to the data, and other data management operations. Storage management system 110 can perform data access or other data management operations in response to requests from clients or users. Storage system 120 may include one or more storage devices to provide storage space for data. These storage devices may be centralized at a single site or distributed across different locations.

[0024] Storage system 120 stores multiple files 122. Storage system 120 can provide access to the stored files 122 based on directories and inodes. Accessing a specific file 122 in storage system 120 will require multiple I / O accesses. For example, when reading a file 122, at least the directory entry corresponding to that file 122 needs to be read into memory. The directory entry records the filename of file 122 and the identifier of its corresponding inode. Then, the inode of file 122 needs to be read into memory. The inode indicates the storage address of file 122 in storage system 120. Finally, the actual content of file 122 is read from storage system 120. During the reading of file 122, information used for file indexing, such as directory entries and inodes (sometimes called "metadata"), needs to be read multiple times.

[0025] Currently, various applications are being developed and becoming increasingly popular, including artificial intelligence, the Internet of Things (IoT), smart finance, smart security, sensor networks, and various scientific and research experiments. These applications generate large amounts of data of different types and sizes that need to be stored. This data includes a large number of small files, meaning that the data volume of each individual file is relatively small. In some application scenarios, the number of small files to be stored can reach tens of millions, billions, or even tens of billions.

[0026] For example, financial applications not only need to scan numerous bills into image files and extract descriptive information to form documents, but also need to store electronic contracts, signature data, facial recognition data, and so on. These files are mostly small, ranging from a few bytes to several hundred bytes, and their number is very large and growing year by year. Furthermore, these small files may need to be stored for a long time, and most require multiple reads after being written once.

[0027] Current storage systems offer high resilience and availability, high storage efficiency, and overall high performance through their storage and access mechanisms. However, such mechanisms present challenges when accessing (e.g., reading and writing) small files. For instance, during file reading, directory entries and inodes can be cached in memory. This reduces the number of I / O accesses to the storage system (e.g., only one file read is needed). Furthermore, access efficiency is improved due to the faster access speed of memory. However, when storing and accessing a large number of small files, the sheer number of files and limited memory space make it impossible to cache all directory entries and inodes in memory. Consequently, for each small file, multiple I / O accesses to the storage system are required to complete the file read. This long-tail effect leads to low overall access efficiency.

[0028] Therefore, it is desirable to provide a solution to improve the access efficiency of storage systems, especially for accessing a large number of small files.

[0029] According to an example embodiment of this disclosure, an improvement is provided. According to this scheme, multiple files are merged into a large file (referred to as a "merged file"). The merged file is stored in a storage system. Thus, access to relatively small multiple files (e.g., read and write operations) is mapped to access to the large merged file, without needing to perform operations related to the index information of each small file. For each small file, a mapping between characteristic values ​​and index information is established. Characteristic values ​​are determined from the filename, and the corresponding index information for the file is determined from the mapping using these characteristic values. The index information is used to index the small file from the large file. Therefore, access to a large number of small files can be converted into access to the large file. Moreover, since the number of files stored in the storage system is reduced, the index information of these files can be loaded into memory, thereby greatly reducing the number of I / O accesses to the storage system and improving overall access efficiency.

[0030] Embodiments of this disclosure will be described in more detail below with reference to the accompanying drawings.

[0031] Figure 2 An example storage environment 200 according to some embodiments of the present disclosure is shown. For example... Figure 2 As shown, the storage environment 200 includes a storage management system 210 and a storage system 220.

[0032] Storage management system 210 is configured to control and manage storage system 220, including storing data in storage system 220, providing access to the data, and other data management operations. Storage management system 210 can perform data access or other data management operations in response to requests from clients or users. Here, "access" to data or files can at least include reading, writing, or deleting data or files.

[0033] Storage management system 210 can be implemented on one or more computing devices, servers, or computing systems. Storage management system 210 can control and manage storage system 220 based on various storage technologies. Storage system 220 may include one or more storage devices (e.g., disks) to provide storage space for data. These storage devices may be centralized at a single site or distributed across different locations. An example of storage system 220 is a Unix file system (UFS), such as a UFS64 system. This is merely an example, and embodiments of this disclosure can also be applied to storage systems based on other storage technologies.

[0034] It should be understood that Figure 2Environment 200 is shown only schematically and is not intended to limit the embodiments of this disclosure. The various systems / devices in environment 200 can be arranged in any other manner. For example, although shown as separate from storage system 220, storage management system 210 and storage system 220 can also be integrated into a single system.

[0035] As briefly mentioned above, according to embodiments of this disclosure, multiple files are merged into one file for storage in a storage system. The merged file is referred to herein as a merged file, and the multiple files being merged are sometimes referred to as smaller files or subfiles. The merged file comprises multiple sections, each section storing one of the merged files. As discussed in detail below, each section stores at least the file's data, and in some embodiments, also stores other information associated with the file, such as metadata associated with the file.

[0036] like Figure 2 As illustrated schematically, storage system 220 stores merged file 230, which includes multiple segments 232-1, 232-2, 232-3, ..., etc. For ease of discussion, segments 232-1, 232-2, 232-3, etc., in merged file 230 are collectively referred to or individually as segment 232. Each segment in merged file 230 corresponds to one of the multiple files being merged. It should be understood that although... Figure 2 The illustration shows at least three files, but the actual merged file 230 may contain more or fewer files. Furthermore, although only one merged file is shown, multiple similar merged files may be stored in the storage system 220. Embodiments of this disclosure are not limited in these respects.

[0037] Furthermore, according to embodiments of this disclosure, for the merged file 230, a mapping 240 between characteristic values ​​and index information is maintained for indexing multiple files from the merged file 230 to enable access to individual files. The mapping 240 may include multiple entries, each corresponding to one of the multiple files in the merged file 230. Each entry includes the characteristic value and index information of the corresponding file.

[0038] The index information in map 240 is used to index the individual files in merged file 230. The filenames of the individual files in merged file 230 are used to determine the characteristic value in map 240. The characteristic value is used to identify a specific file among the multiple files used for merging. Various techniques can be used to calculate the characteristic value from the filename. Filenames can typically be represented as strings. One method for calculating characteristic values ​​is a hash algorithm. For example, a hash function can be applied to the filename, and the resulting hash value can be used to determine the characteristic value. Various hash functions exist that can be applied to calculate hash values. In this paper, the characteristic value is sometimes referred to as a digital digest, and in the case of applying a hash function, it is also called a hash value. In addition to hash values, other algorithms can be used to calculate filename-specific characteristic values ​​from the filename, as long as the calculated characteristic value can be used to identify the filename (and thus the file).

[0039] Figure 3 and Figure 4 Examples of merging files 230 and mapping characteristic values ​​to index information 240 are given respectively. Figure 3 As shown, the merged file 230 includes multiple segments 232-1, 232-2, 232-3, etc., which are used to store the individual files being merged. In some embodiments, the merged file 230 also includes a super block 310, which stores metadata associated with the merged file 230. In one example, the super block 310 may be located at the header of the merged file 230.

[0040] Figure 3 A specific example of the information included in section 232 is also shown. In addition to field 331, which stores file data for individual files, such as... Figure 3 As shown, segment 232 can also be used to store other fields of metadata associated with the file. In some embodiments, segment 232 may also include a data size field 330 for indicating the size of field 331; a special flag header field 321 for identifying segment 232, such as a magic number; a cyclic redundancy check (CRC) field 322 for storing check information for verifying the integrity of file data 331; a total size field 323 for indicating the total size of segment 232; and a characteristic value field 324 for storing the characteristic value of the filename of the file corresponding to segment 232.

[0041] Alternatively or additionally, segment 232 may also include a flag field 325 for marking the deletion status of segment 232, which will be discussed in detail in some embodiments below. Alternatively or additionally, segment 232 may also include a filename length field 326 for indicating the length of the filename corresponding to segment 232; and a filename field 327 for storing the filename of the corresponding file. Alternatively or additionally, segment 232 may also include an optional data length field 328 for indicating the length of optional data; and an optional data field 329 for storing optional data for the corresponding file, such as file creation time, author, access permissions, globally unique identifier (GUID), and other file-related information. Alternatively or additionally, segment 232 may also include a padding field 332 for including padding data if necessary to align the total size of segment 232 to a specific unit required by storage system 220 (e.g., a 512-byte unit), etc.

[0042] Note that since the filenames of the files to be merged have different lengths, different optional data to be stored, and different file data sizes, the total size of the segments 232 allocated to each file in the merged file 230 may also be different.

[0043] It should be understood that Figure 3 Only one specific example of section 232 of merged file 230 is shown. In other embodiments, Figure 3 One or more fields shown in section 232 may be removed from section 232, or additional fields may be added to section 232 to store information associated with the files to be merged. For example, in some cases, section 232 may even include only field 331 for storing file data. Furthermore, the fields in section 232 may be arranged in other orders.

[0044] exist Figure 4 In the example, the mapping 240 between characteristic values ​​and index information includes multiple entries 410-1, 410-2, 410-3, ... etc. For ease of discussion, these multiple entries 410-1, 410-2, 410-3, etc., are collectively referred to as or individually as entry 410. Each entry 410 corresponds to one of the multiple files being merged in the merged file 230, and consequently, each corresponds to one of the multiple sections 232 in the merged file 230. For example, entry 410-1 corresponds to section 232-1, entry 410-2 corresponds to section 232-2, entry 410-3 corresponds to section 232-3, and so on.

[0045] Each entry 410 includes at least a characteristic value and index information for the corresponding file. As mentioned above, the characteristic value is determined based on the filename and is used to identify the corresponding file from among the merged files, while the index information is used to index the corresponding file from the merged file 230. In some embodiments, the index information includes the offset and total size of the segment 232 of the corresponding individual file within the merged file 230. Based on the offset and total size, a specific segment 232 can be located from the merged file 230. In some embodiments, each entry may also include a flag field to mark the deletion status of the corresponding segment 232, which will be discussed in detail in some embodiments below.

[0046] In some embodiments, during storage management, a specific directory for the merged file 230 may be created in the root directory, for example, a directory for a large number of small files. This directory is dedicated to storing small files. Each such directory points to a merged file 230. In the initial stage, the creation of the directory will trigger the creation of the merged file 230 in the storage system 220, which initially has no sections for files. As files are continuously merged into the merged file 230, the size of the merged file can continuously increase. This will be discussed in more detail in the file writing process below.

[0047] Figure 4 Each entry 410-1, 410-2, and 410-3 includes fields for storing characteristic values, index information, and flag bits. As shown, entry 410-1 includes a characteristic value field 412-1, representing the characteristic value of the corresponding file's filename; an offset field 414-1 and a total size field 416-1, storing the offset and total size of the corresponding segment 232 within the merged file 230, respectively, which together serve as index information. Furthermore, entry 410-1 also includes a flag bit field 418-1, used to mark the deletion status of the corresponding segment 232. Similarly, entry 410-2 includes a characteristic value field 412-2, an offset field 414-2, a total size field 416-2, and a flag bit field 418-2; entry 410-3 includes a characteristic value field 412-3, an offset field 414-3, a total size field 416-3, and a flag bit field 418-3, and so on.

[0048] In the following text, for ease of discussion, characteristic value fields 412-1, 412-2, and 412-3 are sometimes collectively referred to as characteristic value field 412 or referred to individually as offset field 414-1, 414-2, and 414-3 are sometimes collectively referred to as offset field 414 or referred to individually as offset field 414, total size fields 416-1, 416-2, and 416-3 are sometimes collectively referred to as total size field 416 or referred to individually as total size field 416, and flag fields 418-1, 418-2, and 418-3 are sometimes collectively referred to as flag field 418 or referred to individually as flag field 418.

[0049] The merged file 230 and mapping 240 are used together to enable access to the individual smaller files within the merged file 230. Mapping 240 can be updated as the merged file 230 is updated to maintain synchronization with it. In a file reading scenario, if a client wants to read a file from the merged file 230, the storage management system 210 can determine the characteristic value based on the file's filename and look up the index information mapped to the determined characteristic value in mapping 240, such as the offset 414 and total size 416 of the corresponding segment 232. Based on the index information, the storage management system 210 reads the corresponding segment 232 from the storage system 220 and returns the stored file data to the client.

[0050] In some embodiments, to further improve access to the individual files in the merged file 230, the mapping 240 may be stored in a storage device with a faster access speed than the disk in the storage system 220. This results in faster access to the mapping 240. In one embodiment, such as Figure 2 As shown, mapping 240 is cached in memory 250. Such mapping 240 can be called in-memory mapping 240. In the mapping from feature values ​​to index information, the amount of information that needs to be maintained for each small file in merged file 230 is small. Therefore, even if the number of small files increases (whether in a single merged file or multiple merged files), the storage space required to store the mapping is acceptable. For example, for each small file in merged file 230, the entry 410 of mapping 240 can be a few bytes in size. For billions of small files, the total size of the mapping is approximately a few gigabytes, which is acceptable in memory cache. Therefore, it is feasible to use storage devices with higher access speeds (e.g., memory) to store the mapping in practical applications.

[0051] If mapping 240 is stored in memory, it may be erased during system power outages, restarts, or other events. In such cases, mapping 240 can be reconstructed for storage in memory 250. In some embodiments, mapping 240 can be regenerated by reading the merged file 230 from storage system 220. In some cases, due to the large total size of the merged file 230, the overhead of reading the merged file 230 and generating mapping 240, including time consumption, will be significant.

[0052] In some embodiments, to accelerate the reconstruction of mapping 240, an index file may also be stored in storage system 220 to store information required for regenerating mapping 240. This information may come from merged file 230 or be calculated based on information therein, but does not need to include all information in merged file 230, such as at least the file data therein. The persistent storage of the index file ensures that the indexes of the individual small files in merged file 230 are not lost. Therefore, the index file can be considered a checkpoint for mapping 240 to help restore mapping 240 at any time. Since the index file is smaller than merged file 230, reading the index file from storage system 220 to regenerate mapping 240 will be faster and more efficient, reducing the initialization time of mapping 240.

[0053] Figure 5 An example storage environment 200 in such an embodiment is shown, wherein an index file 530 is stored in a storage system 220. The index file 530 includes multiple entries 532-1, 532-2, 532-3, ..., etc. For ease of discussion, the multiple entries 532-1, 532-2, 532-3, etc., are collectively referred to or individually as entry 532. The multiple entries 532 correspond to the multiple files being merged in the merged file 230, and consequently also to the multiple sections 232 in the merged file 230. For example, entry 532-1 corresponds to section 232-1, entry 532-2 corresponds to section 232-2, entry 532-3 corresponds to section 232-3, and so on.

[0054] Figure 6 A specific example of the information included in index file 530 is shown. Figure 6 As shown, index file 530 includes a super block 610, which stores metadata associated with index file 530. In one example, super block 610 may be located at the header of index file 530.

[0055] Figure 6 An example of the information included in entry 532 is also shown. Figure 6As shown, entry 532 may include a characteristic value field 621, used to store the characteristic value of the filename corresponding to segment 232. The characteristic value stored in characteristic value field 621 may be the same as the characteristic value field 324 of the corresponding segment 232 in the merged file 230. Entry 532 may also include a flag field 622, used to mark the deletion status of segment 232. The flag in flag field 622 may be the same as the flag field 325 of the corresponding segment 232 in the merged file 230. Entry 532 may also include an offset field 623 and a total size field 624, storing the offset and total size of the corresponding segment 232 in the merged file 230, respectively, to be used together as index information. The offset and total size may be determined based on the position and total size of the corresponding segment 232 in the merged file 230.

[0056] Alternatively or additionally, entry 532 may also include a size field 624 to indicate the size of entry 532. Alternatively or additionally, entry 532 may also include a filename field 625 to store the filename of the file stored in the corresponding segment 232; and a filename length field 626 to indicate the length of the filename stored in filename field 625. The information stored in filename field 625 and filename length field 626 may be the same as that in filename field 327 and filename length field 326 in merged file 230.

[0057] Alternatively or additionally, entry 532 may also include a padding field 627 for including padding data, if necessary, to align the total size of entry 532 to a specific unit required by storage system 220 (e.g., a unit of 512 bytes), etc.

[0058] It should be understood that Figure 6 Only one specific example of entry 532 of index file 530 is shown. In other embodiments, Figure 6 One or more fields shown in the diagram may be removed from entry 532, or additional fields may be added to entry 532 to store information required to generate mapping 240. For example, in some cases, entry 532 may include only the characteristic value field 621, the offset field 623, and the total size field 623, while other fields may be omitted. In some cases, entry 532 may additionally include a flag field 622. In some embodiments, storing the filename itself in entry 532 may help verify the characteristic value in the characteristic value field 621. Furthermore, the fields in entry 532 may be arranged in a different order.

[0059] As mentioned above, index file 530 can be used to reconstruct mapping 240. In some embodiments, the reconstruction of mapping 240 can be performed by storage management system 210. If storage management system 210 determines that mapping 240 has been removed from memory 250, storage management system 210 reads index file 530 from storage system 220. The access method of index file 530 can be similar to the access method provided by storage system 220 for general files. After reading index file 530, storage management system 210 can regenerate mapping 240 based on the characteristic values ​​and index information included in multiple entries 532 in index file 530. Storage management system 210 can cache the regenerated mapping 240 in memory 250 for subsequent access to the individual files in merged file 230.

[0060] In some embodiments, the file access process proposed in this disclosure also facilitates directory traversal. Conventional methods require traversing all directories within a single directory. However, according to embodiments of this disclosure, directory traversal can be provided by reading an index file 530, since the index file 530 stores the filenames of the files in the merged file 230.

[0061] During operation, if subfiles in merged file 230 are added or deleted, index file 530 can also be added or deleted accordingly to reflect the updated merged file 230. In some embodiments, index file 530 may be updated asynchronously with merged file 230. For example, if a new file is merged into merged file 230, merged file 230 will add a new segment 232. Entries 532 in index file 530 may be added asynchronously. As another example, if a file in merged file 230 is deleted, the flag in flag field 325 of the corresponding segment 232 can be set to deleted to indicate the deletion status. However, deletion flag field 622 in index file 530 may be set asynchronously. Such asynchronous updates ensure that write and delete operations on individual files are completed quickly because no additional synchronous disk write operations are required during the execution of file write and delete operations.

[0062] However, asynchronous updates will cause index file 530 to not always reflect the latest merged file 230 in a timely manner. If the mapping 240 is reconstructed using index file 530, which is out of sync with merged file 230, incorrect mapping information will be obtained, thus affecting subsequent file access.

[0063] In some embodiments, before reconstructing the mapping 240 using the index file 530, the storage management system 210 may update the index file 530 to synchronize it with the merged file 230. In some embodiments, the order of multiple entries 532 in the index file 530 may be maintained the same as the order of multiple files in the merged file 230. In some embodiments, during the creation and modification of the merged file 230, a new segment 232 for storing the files to be merged is appended to an existing segment 232 in the merged file 230. If the index file 530 is synchronized to the merged file 230 asynchronously, entries 532 corresponding to new segments 232 may also be appended to existing entries 532.

[0064] Therefore, when updating the index file 530, the storage management system 210 can determine whether the merged file 230 includes files added after the current version of the index file 530 by comparing the merged file 230 with the index file 530. For example, by sequentially comparing existing segments 232 in the merged file 230 with existing entries 532 in the index file 530, the storage management system 210 determines that for one or more segments 232 in the merged file 230, there is no corresponding entry 532 in the index file 530. In this case, the storage management system 210 determines that the segment after the last segment 232 with the corresponding entry 532 is a newly added segment. A segment 232 without a corresponding entry 532 can be called an "orphan" segment 232. The storage management system 210 can quickly identify such "orphan" segments because the last entry 532 in the index file 530 always corresponds to the last "non-orphan" segment in the merged file 230.

[0065] After identifying one or more "orphan" segments 232, the storage management system 210 can determine the characteristic values ​​of one or more files corresponding to the one or more "orphan" segments and the index information used to index these one or more files respectively from the merged file 230. For example, the storage management system 210 can read the "orphan" segments 232 from the storage system 220 and determine the characteristic values ​​and index information based on the read "orphan" segments 232. For example, the characteristic value field 324 of these segments 232 indicates the characteristic values ​​of the corresponding files, and the index information is determined by analyzing the total size and relative position of the "orphan" segments 232. The storage management system 210 can create new entries 532 in the index file 530 to store the determined characteristic values ​​and index information. The storage management system 210 can use the updated index file 530 to generate a mapping 240.

[0066] In some embodiments, the deletion of segment 232 in merged file 230 may initially cause the flag in flag field 325 of segment 232 to be set to deleted, but segment 232 remains in merged file 230. Therefore, when updating index file 530, storage management system 210 may read segment 232 marked as deleted from storage system 220. In this case, storage management system 210 can determine whether the flag in flag field 325 of the read segment 232 is set to deleted. If the flag in flag field 325 of a segment 232 is set to deleted, it means that the corresponding file has been deleted, and storage management system 210 can update mapping 240 accordingly, no longer including entry 410 corresponding to the deleted file (or segment 232) in mapping 240.

[0067] The foregoing has discussed embodiments of file and index information according to some embodiments of this disclosure. The following will describe, with reference to flowcharts, an example process for performing access to small files in a merged file and for reclaiming storage space within the storage environment of this disclosure.

[0068] Figure 7 A flowchart of a file reading process 700 according to some embodiments of the present disclosure is shown. Process 700 can be implemented in... Figure 2 The storage management system 210 is used. For ease of discussion, process 700 will be described from the perspective of the storage management system 210.

[0069] In box 710, storage management system 210 obtains a read request for a target file (sometimes referred to as the "first target file"), the read request including the target filename. For example, the read request may indicate the path to the directory of the target file, such as filename: / tmp / dir / abcdefgh, where "abcdefgh" is the filename. In box 720, storage management system 210 determines a target characteristic value for the target file based on the target filename. For example, the target characteristic value is calculated based on the filename "abcdefgh".

[0070] In box 730, storage management system 210 determines whether the determined characteristic value exists in mapping 240. Storage management system 210 can access mapping 240, for example, from memory 250. If the determined characteristic value exists in mapping 240, in box 740, storage management system 210 determines target index information for the target file to be read, i.e., index information in mapping 240 mapped to the determined characteristic value. Target index information may include, for example, the offset and total size of the segment 232 of the target file within the merged file 230.

[0071] In box 750, storage management system 210 reads the target files included in merged file 230 from storage system 220 based on target index information. For example, storage management system 210 may read the corresponding segment 232 from merged file 230 based on target index information and extract the file data therein. In some embodiments, storage management system 210 may also determine the integrity of file data based on checksum information, such as CRC, in segment 232.

[0072] In some embodiments, if it is determined in block 730 that the target characteristic value does not exist in map 240, the storage management system 210 may use the "target file does not exist" indication as an indication for a read request. In some embodiments, if the determined characteristic value exists in map 240, the storage management system 210 may also check the flag field 418 in entry 410 of map 240. If the flag bit in flag field 418 is set to deleted, the storage management system 210 may not perform a read operation on segment 232, but instead use the "target file has been deleted" indication as an indication for a read request.

[0073] Figure 8 A flowchart of a process 800 for writing a file according to some embodiments of the present disclosure is shown. Process 800 can be implemented in... Figure 2 The storage management system 210 is used. For ease of discussion, process 800 will be described from the perspective of the storage management system 210.

[0074] In box 810, storage management system 210 obtains a write request for a target file (sometimes referred to herein as a "second target file"). The write request may include file data and filename of the target file. In box 820, storage management system 210 determines target characteristic values ​​for the target file based on the target filename.

[0075] At box 830, storage management system 210 determines whether the determined characteristic value exists in mapping 240. Storage management system 210 can access mapping 240, for example, from memory 250. If the determined characteristic value does not exist in mapping 240, meaning the filename is new, then at box 840, storage management system 210 appends a segment 232 for storing the target file to be written after merging one or more existing segments 232 of file 230. Storage management system 210 can allocate storage space in storage system 220 for the new segment 232. Segment 232 stores the file data of the target file to be written and, in some embodiments, also includes other associated metadata, as referenced above. Figure 3 The example described.

[0076] In box 850, storage management system 210 updates mapping 240. Specifically, storage management system 210 adds a new entry 410 to mapping 240 to store characteristic values ​​for the target file being written and index information in merged file 230. In some embodiments, index file 530 can be updated asynchronously to append a new entry 532 after the existing entry 532 to store information for the target file being written. The format of the new entry 532 can be referenced. Figure 4 Examples are provided. In some cases, synchronous updates to index file 530 can also be performed, and the embodiments of this disclosure are not limited in this respect.

[0077] In some embodiments, if it is determined in block 830 that mapping 240 has a determined property value, which means that the target file to be written has the same name as an existing file in merged file 230, storage management system 210 can provide an indication that "filename already exists" as a response to the write request.

[0078] In some embodiments, when writing files, the storage management system 210 can also constrain the writing of smaller files to the merged file 230 by setting a threshold. For example, the storage management system 210 can determine whether the size of the target file to be written is lower than a threshold size. The threshold size can be set as needed, for example, it can be 1MB, 2MB, etc. If the size of the target file to be written is lower than the threshold size, the storage management system 210 can perform the file writing process 800. Otherwise, the storage management system 210 can store the target file in a conventional file writing manner. In this way, the file access process proposed in the embodiments of this disclosure can be compatible with existing file access processes. The file access process proposed in the embodiments of this disclosure can be used to perform access to relatively small files.

[0079] Figure 9 A flowchart of a process 900 for file deletion according to some embodiments of the present disclosure is shown. Process 900 can be implemented in... Figure 2 The storage management system 210 is used. For ease of discussion, process 900 will be described from the perspective of storage management system 210.

[0080] In box 910, storage management system 210 receives a deletion request for a target file (sometimes referred to herein as a "third target file"), the request including the target filename. In box 920, storage management system 210 determines target characteristic values ​​for the target file based on the target filename.

[0081] In box 930, storage management system 210 determines whether mapping 240 contains the determined characteristic value. Storage management system 210 can access mapping 240, for example, from memory 250. If mapping 240 contains the determined characteristic value, in box 940, storage management system 210 sets the flag bit for the target file in mapping 240 to deleted. For example, based on the determined characteristic value, storage management system 210 locates the corresponding entry 410 in mapping 240 and sets the flag bit in flag field 418 of entry 410 to deleted.

[0082] If the determined characteristic value exists in mapping 240, in box 950, storage management system 210 can also set the flag bit for the target file in merged file 230 to deleted. For example, based on the determined characteristic value, storage management system 210 finds the corresponding segment 232 in merged file 230 and sets the flag bit in flag field 325 of segment 232 to deleted. Note that the execution order of boxes 940 and 950 can be interchanged or executed in parallel.

[0083] In some embodiments, the index file 530 may be updated asynchronously to set the flag in the flag field 622 of the corresponding entry 532 to deleted. In some cases, synchronous updates to the index file 530 may also be performed, and the embodiments of this disclosure are not limited in this respect.

[0084] exist Figure 9 In an example embodiment, file deletion causes a flag in segment 232 of merged file 230 to be set as deleted, but segment 232 may still occupy storage space in storage system 220. In some embodiments, storage management system 210 may also perform storage space reclamation. The storage space reclamation process may be performed periodically or triggered by other events (e.g., based on client / user requests).

[0085] Figure 10 A flowchart of a storage space reclamation process 1000 according to some embodiments of the present disclosure is shown. Process 1000 can be implemented in... Figure 2 The storage management system 210 is used. For ease of discussion, process 1000 will be described from the perspective of the storage management system 210.

[0086] When reclaiming storage space, the storage management system 210 can specify another merge file in the storage system 220 to migrate the segments 232 corresponding to the files that have not been deleted in the merge file 230 to the other merge file.

[0087] Specifically, in box 1010, storage management system 210 determines whether multiple flag bits corresponding to one or more files in merged file 230 are set to deleted. Storage management system 210 can determine whether one or more files are deleted by checking the flag bit field 325 of each segment 232 in merged file 230. If at least one flag bit corresponding to at least one file in merged file 230 is not set to deleted, in box 1020, storage management system 210 copies the at least one file that was not deleted to another merged file in storage system 220. The other file can be newly created or an existing merged file.

[0088] In box 1030, storage management system 210 deletes merged file 230 from storage system 220 to reclaim storage space for merged file 230. Storage management system 210 can block write requests for merged file 230 after copying from segment 232 to another merged file is complete, and then delete merged file 230 from storage system 220 to reclaim storage space.

[0089] In box 1040, storage management system 210 updates mapping 240 and index file 530 based on another merged file. If the other merged file is a previously created merged file, storage management system 210 can use mapping 240 to update the mapping corresponding to the other merged file. In this example, storage management system 210 can modify the index file corresponding to the other merged file and can delete index file 530. If the other merged file is a newly created merged file for storage space reclamation, storage management system 210 can update mappings 240 and 530 accordingly to reflect any files that were not deleted in the other merged file.

[0090] In some embodiments, if one or more files in merged file 230 are deleted during the copying process from merged file 230 to another merged file, in order to ensure file state synchronization, the storage management system 210 may also set the flag bit of the flag field 325 of the corresponding segment 232 to deleted while copying the corresponding segment 232 to another merged file.

[0091] It should be understood that although the operations in the process are illustrated and described in a specific order, some or all of these operations may be performed in a different order or in parallel. In some embodiments, certain operations may be omitted, or other operations may be added.

[0092] Figure 11 A block diagram schematically illustrates a device 1100 that can be used to implement embodiments of the present disclosure. It should be understood that... Figure 11The device 1100 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 11 The device 1100 shown can be used to implement Figures 7 to 10 The process is 700 to 1000. Figure 11 The device 1100 shown can be implemented as or included in Figure 2 In the storage management system 210.

[0093] like Figure 11 As shown, device 1100 includes a central processing unit (CPU) 1101, which can perform various appropriate actions and processes according to computer program instructions stored in read-only storage device (ROM) 1102 or loaded from storage unit 1108 into random access storage device (RAM) 1103. Various programs and data required for the operation of device 1100 can also be stored in RAM 1103. CPU 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. Input / output (I / O) interface 1105 is also connected to bus 1104.

[0094] Multiple components in device 1100 are connected to I / O interface 1105, including: input unit 1106, such as keyboard, mouse, etc.; output unit 1107, such as various types of monitors, speakers, etc.; storage unit 1108, such as disk, optical disk, etc.; and communication unit 1109, such as network card, modem, wireless transceiver, etc. Communication unit 1109 allows device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0095] The various processes and procedures described above, such as processes 700 to 1000, may be executed by processing unit 1101. For example, in some embodiments, processes 700 to 1000 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by CPU 1101, one or more steps of processes 700 to 1000 described above may be performed.

[0096] Embodiments of this disclosure may also provide a computer-readable storage medium having computer-executable instructions stored thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions that are executed by a processor to implement the methods described above.

[0097] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, computer-readable media, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0098] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0099] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0100] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0101] As used herein, the term "determine" encompasses a wide variety of actions. For example, "determine" can include operations, calculations, processing, deriving, investigation, searching (e.g., looking in a table, database, or other data structure), ascertaining, etc. Furthermore, "determine" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), etc. Additionally, "determine" can include parsing, selecting, choosing, building, etc.

[0102] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for storage management, comprising: Obtaining a read request for a first target file, the read request including a first target file name of the first target file; Determining a first target characteristic value for the first target file based on the first target file name; Determining first target index information for the first target file from a mapping between characteristic values and index information, the first target index information being used to index the first target file from a merged file, the merged file including multiple segments for storing multiple files and being stored in a storage system, the multiple files including the first target file; Reading the first target file included in the merged file from the storage system based on the first target index information; If it is determined that the mapping is removed from the memory, reading an index file corresponding to the merged file from the storage system, where the mapping is cached in the memory, the index file including multiple entries respectively corresponding to the multiple files, each entry at least including a characteristic value for the corresponding file and index information for indexing the corresponding file from the merged file; Regenerating the mapping based on the characteristic values and index information included in the multiple entries in the index file; And Caching the regenerated mapping in the memory.

2. The method according to claim 1, wherein determining the first target index information includes: Determining the first target index information by accessing the mapping from the memory.

3. The method according to claim 1, wherein the index file is updated asynchronously with the merged file, and the order of the multiple entries in the index file is the same as the order of the multiple files in the merged file, and wherein the method further includes: Determining, by comparing the merged file with the index file, that the merged file further includes at least one file added after the multiple files, Determining at least one characteristic value respectively corresponding to the at least one file and index information for indexing the at least one file from the merged file, and Creating at least one new entry in the index file for storing the determined at least one characteristic value and index information.

4. The method according to claim 1, wherein the target index information includes an offset and a total size of a segment for storing the first target file in the merged file.

5. The method according to claim 1, further comprising: Obtaining a write request for a second target file; And Attaching a segment for storing the second target file after the multiple segments of the merged file.

6. The method according to claim 5, wherein attaching a segment for storing the second target file includes: Determining whether the size of the second target file is less than a threshold size; And If the size of the second target file is less than the threshold size, attaching a segment for storing the second target file after the multiple segments of the merged file.

7. The method according to claim 1, wherein each of the plurality of segments further stores metadata associated with the plurality of files.

8. The method according to claim 1, further comprising: obtaining a deletion request for a third target file, the read request including a third target file name of the third target file; determining a third target characteristic value for the third target file based on the third target file name; setting a first flag bit for the third target file in the mapping to deleted based on the third target characteristic value, so as to prevent access to the third target characteristic value and third target index information mapped to the third target characteristic value; and setting a second flag bit for the third target file in the merged file to deleted, so as to prevent access to the third target file.

9. The method according to claim 1, further comprising: determining whether at least one flag bit corresponding to each of the plurality of files is set to deleted; if at least one flag bit corresponding to each of the at least one file is not set to deleted, copying the at least one file to another merged file stored in the storage system; deleting the merged file from the storage system; and updating the mapping based on the another merged file.

10. The method according to claim 9, further comprising: updating an index file corresponding to the merged file stored in the storage system based on the another merged file, the updated index file including at least one entry corresponding to each of the at least one file, each entry including at least a characteristic value of the corresponding file and index information for indexing the corresponding file from the merged file.

11. An electronic device, comprising: at least one processor; and at least one memory storing computer program instructions, the at least one memory and the computer program instructions being configured to, together with the at least one processor, cause the electronic device to perform actions, the actions including: obtaining a read request for a first target file, the read request including a first target file name of the first target file; determining a first target characteristic value for the first target file based on the first target file name; determining first target index information for the first target file from a mapping of characteristic values and index information based on the first target characteristic value, the first target index information being used to index the first target file from a merged file, the merged file including a plurality of segments for storing a plurality of files and being stored in a storage system, the plurality of files including the first target file; reading the first target file included in the merged file from the storage system based on the first target index information; If it is determined that the mapping has been removed from the memory, read the index file corresponding to the merged file from the storage system, where the mapping is cached in the memory, and the index file includes multiple entries corresponding to each of the multiple files, and each entry at least includes a characteristic value for the corresponding file and index information for indexing the corresponding file from the merged file; Regenerate the mapping based on the characteristic values and index information included in the multiple entries in the index file; and Cache the regenerated mapping in the memory.

12. The device according to claim 11, wherein determining the first target index information includes: Determining the first target index information by accessing the mapping from the memory.

13. The device according to claim 11, wherein the index file is updated asynchronously with the merged file, and the order of the multiple entries in the index file is the same as the order of the multiple files in the merged file, and where the action further includes: Determining, by comparing the merged file with the index file, that the merged file further includes at least one file added after the multiple files, Determining at least one characteristic value corresponding to each of the at least one file and index information for indexing each of the at least one file from the merged file, and Creating at least one new entry in the index file for storing the determined at least one characteristic value and index information.

14. The device according to claim 11, wherein the target index information includes the offset and total size of the section for storing the first target file in the merged file.

15. The device according to claim 11, wherein the action further includes: Obtaining a write request for a second target file; and Attaching a section for storing the section of the second target file after the multiple sections of the merged file.

16. The device according to claim 15, wherein attaching a section for storing the section of the second target file includes: Determining whether the size of the second target file is lower than a threshold size; and If the size of the second target file is lower than the threshold size, attaching a section for storing the section of the second target file after the multiple sections of the merged file.

17. The device according to claim 11, wherein the multiple sections respectively further store metadata associated with the multiple files.

18. The device according to claim 11, wherein the action further includes: Obtaining a deletion request for a third target file, the read request including a third target file name of the third target file; Determining a third target characteristic value for the third target file based on the third target file name; Based on the third target characteristic value, setting a first flag bit for the third target file in the mapping to deleted to prevent access to the third target characteristic value and the third target index information mapped to the third target characteristic value; and Set the second flag bit for the third target file in the merged file to deleted to prevent access to the third target file.

19. The apparatus according to claim 11, wherein the action further comprises: Determining whether the plurality of flag bits respectively corresponding to at least one file in the plurality of files are set to deleted; If at least one of the flag bits respectively corresponding to the at least one file is not set to deleted, copying the at least one file to another merged file stored in the storage system; Deleting the merged file from the storage system; And Updating the mapping based on the other merged file.

20. The apparatus according to claim 19, wherein the action further comprises: Updating an index file corresponding to the merged file stored in the storage system based on the other merged file, the updated index file including at least one entry respectively corresponding to the at least one file, each entry including at least a characteristic value of the corresponding file and index information for indexing the corresponding file from the merged file.

21. A computer program product, the computer program product being tangibly stored on a non-volatile computer-readable medium and comprising computer-executable instructions that, when executed, cause an apparatus to perform the method according to any one of claims 1 to 10.

Images