A data aggregation method, device, distributed storage system and storage medium
Through calculation and utilization of the cost of hollow compression, the hollows of small objects in the distributed storage system are effectively utilized, which solves the problem of low storage space utilization and realizes efficient utilization of storage space.
Patent Information
- Application Number
- CN202411060028.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-02
AI Technical Summary
In a distributed object storage system, the void caused by the deletion or expiration of small objects cannot be effectively utilized, resulting in low storage space utilization and the newly uploaded small objects cannot be directly written into the void.
By calculating the cost of hollow compression, obtain the cost of hollow compression of the pending data aggregation file, write it to the new data aggregation file, and delete the original data aggregation file, thereby optimizing the utilization of storage space.
Improves storage space utilization, consumes minimal resources and frees up as much storage space as possible as possible.
Smart Images

Figure CN118585140B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of distributed data storage, and in particular to a data aggregation method, device, distributed storage system and storage medium. Background Art
[0002] Distributed object storage is a data storage technology that can store data in multiple nodes and access and manage it through the network. In the current distributed object storage system, the object aggregation function has been implemented for small objects, that is, multiple small objects are aggregated into a large object, and the aggregation of small objects that are overwritten has been implemented. Among them, small objects are data with a size less than a certain threshold.
[0003] When users delete small objects or small objects expire, holes will be formed in the storage space occupied by these small objects. Since the size of newly uploaded small objects is often different from the size of deleted small objects, newly uploaded small objects cannot be directly written into the holes, which will result in holes that cannot be effectively utilized and low storage space utilization. Summary of the invention
[0004] The purpose of the embodiments of the present application is to provide a data aggregation method, device, distributed storage system and storage medium to improve storage space utilization. The specific technical solution is as follows:
[0005] In a first aspect, an embodiment of the present application provides a data aggregation method, which is applied to a distributed storage system, wherein the distributed storage system includes at least one data aggregation file, and the data aggregation file is used to aggregate and store multiple data objects. The method includes:
[0006] Obtaining a hole compression cost corresponding to a to-be-processed data aggregate file, wherein the to-be-processed data aggregate file includes valid data objects and holes, and the hole compression cost is used to represent the cost required to migrate the valid data objects to a new storage space to release the holes;
[0007] Write the valid data objects in the to-be-processed data aggregation file whose corresponding hole compression cost is not greater than a preset threshold into a new data aggregation file;
[0008] Delete the to-be-processed data aggregation file whose corresponding hole compression cost is not greater than a preset threshold.
[0009] Optionally, the calculation method of the hole compression cost includes:
[0010] For each data aggregation file, according to the relationship between the data object parameters in the data aggregation file and the cost required to migrate the valid data object to the new storage space to release the hole, calculate the hole compression cost;
[0011] Among them, the data object parameters include at least one of the discrete degree of holes, the number of valid data objects, the valid data object rate and the size of the storage space that can be released by migrating the valid data objects; the valid data object rate is the ratio of the number of valid data objects to the total number of objects included in the data aggregation file, and the total number of objects includes the number of valid data objects and the number of holes.
[0012] Optionally, the calculation method of the discrete degree of the void includes:
[0013] For each data aggregation file, the holes in the data aggregation file are continuously arranged in the order of corresponding storage positions to obtain arranged continuous holes;
[0014] The span between the first preset position and the second preset position in the continuous hole corresponding to the position in the data aggregation file is calculated as the discrete degree of the hole corresponding to the data aggregation file.
[0015] Optionally, the step of calculating the span between the positions corresponding to the first preset position and the second preset position in the continuous hole in the data aggregation file as the discrete degree of the hole corresponding to the data aggregation file includes:
[0016] Normalizing the continuous holes based on the span from the first hole to the last hole in the data aggregation file;
[0017] The span between the first preset position and the second preset position in the normalized continuous hole is determined as the discrete degree of the hole corresponding to the data aggregation file.
[0018] Optionally, the step of calculating the hole compression cost according to the relationship between the data object parameters in the data aggregation file and the cost required to migrate the valid data object to the new storage space to release the hole includes:
[0019] Based on the positive correlation or negative correlation between the data object parameter and the cost required to migrate the valid data object to the new storage space to release the hole, the hole compression cost corresponding to the data aggregation file is calculated;
[0020] Among them, the discreteness of the hole, the number of valid data objects and the valid data object rate are positively correlated with the cost of migrating the valid data objects to a new storage space to release the hole, and the size of the storage space that can be released by migrating the valid data objects is negatively correlated with the cost of migrating the valid data objects to a new storage space to release the hole.
[0021] Optionally, the step of calculating the hole compression cost corresponding to the data aggregation file based on the positive correlation or negative correlation between the data object parameter and the cost required to migrate the valid data object to the new storage space to release the hole includes:
[0022] In a case where the data object parameters include the discrete degree of the hole, the number of the valid data objects, the valid data object rate, and the size of the storage space that can be released by migrating the valid data objects, calculating a ratio between a product of a proportional parameter included in the data object parameters and an inverse parameter included in the data object parameters as a hole compression cost corresponding to the data aggregation file;
[0023] The proportional parameters include the discrete degree of the holes, the number of the valid data objects and the valid data object rate, and the inverse parameters include the size of the storage space that can be released by migrating the valid data objects.
[0024] Optionally, the data aggregation file has a corresponding index aggregation file, and the index aggregation file is used to store metadata of data objects included in the corresponding data aggregation file;
[0025] The method further comprises:
[0026] Obtaining a data object deletion instruction, and determining an index aggregation file corresponding to the data object to be deleted indicated by the data object deletion instruction;
[0027] Setting the deletion mark corresponding to the metadata of the data object to be deleted in the determined index aggregation file to a preset value, wherein the metadata corresponding to the deletion mark is the preset value and the data object corresponding to the metadata in the data aggregation file is invalid data;
[0028] Count the total number of objects in the data aggregation file corresponding to the determined index aggregation file;
[0029] Determine the number of valid data objects in the data aggregation file corresponding to the index aggregation file according to the number of deletion marks with a preset value in the determined index aggregation file;
[0030] Based on the total number and the number of valid data objects in the data aggregation file corresponding to the index aggregation file, calculate the hole compression cost corresponding to the data aggregation file corresponding to the index aggregation file;
[0031] The total number, the number of valid data objects in the data aggregation file corresponding to the index aggregation file, and the hole compression cost corresponding to the data aggregation file corresponding to the index aggregation file are recorded in the extended attributes of the index aggregation file.
[0032] In a second aspect, an embodiment of the present application provides a data aggregation device, which is applied to a distributed storage system, wherein the distributed storage system includes at least one data aggregation file, and the data aggregation file is used to aggregate and store multiple data objects. The device includes:
[0033] A hole compression cost acquisition module, used to acquire a hole compression cost corresponding to a data aggregate file to be processed, wherein the data aggregate file to be processed includes valid data objects and holes, and the hole compression cost is used to represent the cost required to migrate the valid data objects to a new storage space to release the holes;
[0034] A valid data object writing module is used to write the valid data objects in the to-be-processed data aggregation file whose corresponding hole compression cost is not greater than a preset threshold into a new data aggregation file;
[0035] The file deletion module is used to delete the to-be-processed data aggregation file whose corresponding hole compression cost is not greater than a preset threshold.
[0036] In a third aspect, an embodiment of the present application provides a distributed storage system, including:
[0037] Memory, used to store computer programs;
[0038] The processor is used to implement any method described in the first aspect when executing a program stored in the memory.
[0039] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any method described in the first aspect is implemented.
[0040] In the scheme provided in the embodiment of the present application, it is applied to a distributed storage system, the distributed storage system includes at least one data aggregation file, the data aggregation file is used to aggregate and store multiple data objects, and the distributed storage system can obtain the hole compression cost corresponding to the data aggregation file to be processed, wherein the data aggregation file to be processed includes valid data objects and holes, the hole compression cost is used to represent the cost required to migrate the valid data objects to a new storage space to release the holes, write the valid data objects in the data aggregation file to be processed whose corresponding hole compression cost is not greater than a preset threshold into the new data aggregation file, and delete the data aggregation file to be processed whose corresponding hole compression cost is not greater than the preset threshold. Since the hole compression cost can represent the cost required to migrate the valid data objects to the new storage space to release the holes, then when the hole compression cost corresponding to the data aggregation file to be processed is obtained, the data aggregation files with smaller hole compression costs can be aggregated, that is, the data aggregation files with hole compression costs not greater than the preset threshold are aggregated, and the original data aggregation files are deleted after aggregation, so that the data aggregation files with smaller hole compression costs are preferentially selected for hole compression, which can achieve the purpose of consuming the least resources and releasing as much storage space as possible as quickly as possible, thereby improving the storage space utilization rate. Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.
[0042] Figure 1 A flowchart of a data aggregation method provided in an embodiment of the present application;
[0043] Figure 2 A schematic diagram of data aggregation file aggregation provided in an embodiment of the present application;
[0044] Figure 3 Another schematic diagram of data aggregation file aggregation provided in an embodiment of the present application;
[0045] Figure 4 A flow chart for calculating the discreteness of a hole provided in an embodiment of the present application;
[0046] Figure 5 A schematic diagram of calculating the discrete degree of voids provided in an embodiment of the present application;
[0047] Figure 6A schematic diagram of a data aggregation file provided in an embodiment of the present application;
[0048] Figure 7 A schematic diagram of an index aggregation file provided in an embodiment of the present application;
[0049] Figure 8 A flow chart of writing data objects provided by an embodiment of the present application;
[0050] Fig. 9 A signaling diagram for uploading data objects provided in an embodiment of the present application;
[0051] Fig.10 A schematic diagram of a storage path provided in an embodiment of the present application;
[0052] Fig.11 A flowchart of deleting a data object provided by an embodiment of the present application;
[0053] Fig.12 A signaling diagram for deleting a data object provided in an embodiment of the present application;
[0054] Fig.13 A specific flow chart of the data aggregation method provided in the embodiment of the present application;
[0055] Fig.14 A schematic diagram of a data aggregation device provided in an embodiment of the present application;
[0056] Fig.15 A schematic diagram of a distributed storage system provided in an embodiment of the present application. DETAILED DESCRIPTION
[0057] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.
[0058] In order to improve storage space utilization, the embodiments of the present application provide a data aggregation method, device, distributed storage system, computer-readable storage medium, and computer program product. The following first introduces a data aggregation method provided by the embodiments of the present application.
[0059] like Figure 1 As shown, a data aggregation method is applied to a distributed storage system, wherein the distributed storage system includes at least one data aggregation file, and the data aggregation file is used to aggregate and store multiple data objects. The method includes:
[0060] S101, obtaining the hole compression cost corresponding to the data aggregation file to be processed;
[0061] The data aggregation file to be processed includes valid data objects and holes, and the hole compression cost is used to represent the cost required to migrate the valid data objects to a new storage space to release the holes;
[0062] S102, writing the valid data objects in the to-be-processed data aggregation file whose corresponding hole compression cost is not greater than a preset threshold into a new data aggregation file;
[0063] S103, deleting the to-be-processed data aggregation file whose corresponding hole compression cost is not greater than a preset threshold.
[0064] It can be seen that in the solution provided in the embodiment of the present application, since the hole compression cost can represent the cost required to migrate valid data objects to new storage space to release holes, when the hole compression cost corresponding to the data aggregation file to be processed is obtained, the data aggregation files with smaller hole compression costs can be aggregated, that is, the data aggregation files with a hole compression cost not greater than a preset threshold are aggregated, and the original data aggregation files are deleted after aggregation. In this way, data aggregation files with smaller hole compression costs are preferentially selected for hole compression, which can achieve the purpose of consuming the least resources and releasing as much storage space as possible as quickly as possible, thereby improving the storage space utilization rate.
[0065] Distributed storage is a data storage technology that uses disk space on multiple machines through the network and forms a virtual storage device with these scattered storage resources. Data is scattered on multiple machines. A distributed storage system is a system based on a distributed architecture, where multiple storage nodes with coordinated software and hardware are interconnected through a network to provide storage services. For example, a distributed file system (DFS) means that the physical storage resources managed by the file system are not necessarily directly connected to the local node, but are connected to the node (such as a computer) through a computer network, or are a complete hierarchical file system formed by combining several different logical disk partitions or volumes. DFS provides a logical tree-like file system structure for resources distributed anywhere on the network, making it easier for users to access shared files distributed on the network.
[0066] Object storage is a technology used to store and manage large-scale data. It can store data in the form of objects in a distributed storage system, obtain data objects, and use unique identifiers to access and manage these data objects. Data objects can include data, metadata, and unique identifiers. Data is the actual stored content, which can be any type of data such as text, pictures, videos, etc. Metadata contains attribute information of data objects, such as size, type, creation time, modification time, etc.; and unique identifiers are used to uniquely identify a data object and access and manage it in the storage system.
[0067] The distributed storage system can use aggregate storage to store multiple data objects, that is, an optimized storage method of aggregating and storing several data objects in one storage unit is used to reduce the number of storage units. In an embodiment of the present application, the distributed storage system aggregates and stores multiple data objects in a data aggregation file, that is, the data aggregation file can be used to aggregate and store multiple data objects. Exemplarily, the distributed file system uses aggregate storage to store valid data objects uploaded by users in one file, which is a data aggregation file.
[0068] Specifically, when a user uploads a data object, the distributed storage system can write the data object into a data aggregation file. If the data aggregation file already includes multiple data objects, the data object uploaded by the user is written after the multiple data objects, that is, the data objects in the data aggregation file can be aggregated and stored according to the writing time. For example, if a user uploads images a.jpg, b.jpg, and c.jpg, the distributed storage system can write a.jpg, b.jpg, and c.jpg into the data aggregation file in sequence for aggregate storage.
[0069] If a user deletes a data object in a data aggregation file or a data object in a data aggregation file expires, the storage space occupied by the deleted data object or the storage space occupied by the expired data object should be released for recycling and reuse. However, since the data aggregation file is a whole and the storage space it occupies is continuous, the storage space occupied by the deleted data object or the storage space occupied by the expired data object will form a hole in the continuous storage space.
[0070] In one embodiment, in order to release holes, valid data objects in a data aggregation file may be migrated to a new storage space. Valid data objects are data objects stored in a data aggregation file that have not been deleted and have not expired. If there are multiple data aggregation files, valid data objects in multiple data aggregation files may be continuously written into a new storage space, that is, valid data objects in multiple data aggregation files may be aggregated, and the original data aggregation files including holes may be deleted after aggregation. However, migrating valid data objects may incur a hole compression cost. If the hole compression cost is too large, that is, the resources and time consumed by migrating valid data objects are too large, then there is no need to migrate the valid data objects. If the hole compression cost is within a certain threshold, then the valid data objects may be migrated.
[0071] Among them, the hole compression cost can be used to represent the cost required to migrate valid data objects to new storage space to release holes. Releasing holes means writing valid data objects to new continuous storage space in sequence and continuously. There are no holes between two adjacent valid data objects, so there are no holes in the new data aggregation file.
[0072] In step S101, the distributed storage system may obtain a hole compression cost corresponding to a to-be-processed data aggregate file, wherein the to-be-processed data aggregate file includes valid data objects and holes.
[0073] In one embodiment, the distributed storage system can calculate the hole compression cost of the data aggregate file to be processed. In another embodiment, the distributed storage system can obtain the hole compression cost of the data aggregate file to be processed from a preset file. The preset file can be a data aggregate file, or it can be other files set by the user, which is not specifically limited here. For example, every time a user deletes a data object in a data aggregate file, or a data object in a data aggregate file expires, the distributed storage system can instantly calculate the hole compression cost of the data aggregate file and store it in the data aggregate file, so that the distributed storage system can directly obtain the hole compression cost from the data aggregate file to be processed.
[0074] In order to achieve the migration of valid data objects while consuming as few resources as possible, the pending data aggregate files whose hole compression costs meet a certain threshold can be aggregated. In one embodiment, when the hole compression cost is obtained, the distributed storage system can write the valid data objects in the pending data aggregate files whose corresponding hole compression costs are not greater than the preset threshold into new data aggregate files, that is, execute step S102. Among them, the new data aggregate file refers to a blank file with continuous storage space and no holes. The preset threshold can be set according to actual conditions and is not specifically limited here. The corresponding pending data aggregate files whose hole compression costs are not greater than the preset threshold may be one or more.
[0075] Then, after the distributed storage system aggregates the corresponding data aggregation files to be processed whose hole compression costs are not greater than the preset threshold, that is, the new data aggregation files store valid data objects, the corresponding data aggregation files to be processed whose hole compression costs are not greater than the preset threshold can be deleted. In this way, the holes are released, that is, there are no holes, and hole compression is achieved.
[0076] For example, Figure 2 As shown, there is one data aggregate file to be processed whose corresponding hole compression cost is not greater than the preset threshold, namely, data aggregate file to be processed 201, and the data aggregate file to be processed includes 3 valid data objects and 2 holes, namely objdata_1, hole 1, objdata_2, objdata_3 and hole 2. Then, objdata_1, objdata_2, and objdata_3 can be written into the new data aggregate file 202 in sequence, and then the data aggregate file to be processed 201 can be deleted, thereby releasing the holes in the data aggregate file to be processed 201.
[0077] For example, Figure 3As shown, there are two data aggregation files to be processed whose corresponding hole compression cost is not greater than the preset threshold, namely, a first data aggregation file to be processed 301 and a second data aggregation file to be processed 302, wherein the first data aggregation file to be processed 301 includes M1 valid data objects, namely, objdata_1 to objdata_M1, and N1 holes, namely, hole 1 to hole N1, and the second data aggregation file to be processed 302 includes M2 valid data objects, namely, objdata_1 to objdata_M2, and N2 holes, namely, hole 1 to hole N2. The distributed storage system can write M1 valid data objects objdata_1 to objdata_M1 in the first data aggregate file 301 to be processed and M2 valid data objects objdata_1 to objdata_M2 in the second data aggregate file 302 to be processed into a new data aggregate file 303, and then delete the first data aggregate file 301 to be processed and the second data aggregate file 302 to be processed, thereby releasing N1 holes in the first data aggregate file 301 to be processed and N2 holes in the second data aggregate file 302 to be processed.
[0078] In the scheme of this embodiment, since the hole compression cost can represent the cost required to migrate valid data objects to new storage space to release holes, when the hole compression cost corresponding to the data aggregate file to be processed is obtained, the data aggregate file with a smaller hole compression cost can be selected for hole compression, that is, the data aggregate file with a hole compression cost not greater than a preset threshold is aggregated. In this way, the data aggregate files with a smaller hole compression cost are aggregated first, which can achieve the purpose of consuming the least resources and releasing the most storage space as quickly as possible, thereby improving the storage space utilization rate.
[0079] As an implementation of the embodiment of the present application, the calculation method of the above-mentioned hole compression cost may include:
[0080] For each data aggregation file, according to the relationship between the data object parameters in the data aggregation file and the cost required to migrate the valid data object to the new storage space to release the hole, calculate the hole compression cost;
[0081] Among them, the data object parameters include at least one of the discrete degree of holes, the number of valid data objects, the valid data object rate and the size of the storage space that can be released by migrating the valid data objects; the valid data object rate is the ratio of the number of valid data objects to the total number of objects included in the data aggregation file, and the total number of objects includes the number of valid data objects and the number of holes.
[0082] Since the hole compression cost represents the cost required to migrate valid data objects to a new storage space to release a hole, the calculation of the hole compression cost may take into account factors related to the hole and the valid data object. In one embodiment, for each data aggregation file, the distributed storage system may calculate the hole compression cost based on the relationship between the data object parameters in the data aggregation file and the cost required to migrate the valid data object to a new storage space to release the hole.
[0083] The cost required to migrate valid data objects to the new storage space to release holes is the cost required to write valid data objects from the original storage space to the new storage space in sequence and continuously. The data object parameters include at least one of the discrete degree of holes, the number of valid data objects, the valid data object rate, and the size of the storage space that can be released by migrating valid data objects. The valid data object rate is the ratio of the number of valid data objects to the total number of objects included in the data aggregation file, and the total number of objects includes the number of valid data objects and the number of holes.
[0084] That is to say, the data object parameters may include only one of the discrete degree of holes, the number of valid data objects, the valid data object rate, and the size of the storage space that can be released by migrating valid data objects, or they may include multiple or all of the discrete degree of holes, the number of valid data objects, the valid data object rate, and the size of the storage space that can be released by migrating valid data objects.
[0085] There is a positive correlation or negative correlation between the data object parameters and the cost required to migrate valid data objects to a new storage space to release holes. As an implementation method, the distributed storage system can calculate the hole compression cost corresponding to the data aggregation file based on the positive correlation or negative correlation between the data object parameters and the cost required to migrate valid data objects to a new storage space to release holes.
[0086] Among them, the discrete degree of holes, the number of valid data objects and the valid data object rate are positively correlated with the cost of migrating valid data objects to new storage space to release holes, and the size of storage space that can be released by migrating valid data objects is negatively correlated with the cost of migrating valid data objects to new storage space to release holes.
[0087] Specifically, the cost of migrating valid data objects to new storage space to release holes is related to the degree of discreteness of the holes. The greater the degree of discreteness of the holes, the more storage space the distributed storage system needs to read, the greater the cost of migrating valid data objects to new storage space to release holes, and the greater the cost of hole compression. It can be seen that the degree of discreteness of holes is positively correlated with the cost of migrating valid data objects to new storage space to release holes. For example, the hole compression cost is proportional to the degree of discreteness of holes.
[0088] The cost required to migrate valid data objects to new storage space to release holes is related to the number of valid data objects. The more valid data objects there are, the more indexes need to be updated. The greater the cost required to migrate valid data objects to new storage space to release holes, the greater the hole compression cost. It can be seen that the number of valid data objects is positively correlated with the cost required to migrate valid data objects to new storage space to release holes. For example, the hole compression cost is proportional to the number of valid data objects.
[0089] The cost required to migrate valid data objects to new storage space to release holes is related to the valid data object rate. The higher the valid data object rate, the smaller the proportion of holes in the data aggregation file, the more valid data objects need to be migrated, the greater the cost required to migrate valid data objects to new storage space to release holes, and the greater the hole compression cost. It can be seen that the valid data object rate is positively correlated with the cost required to migrate valid data objects to new storage space to release holes. For example, the hole compression cost is proportional to the valid data object rate.
[0090] The cost required to migrate valid data objects to a new storage space to release holes is also related to the size of the storage space that can be released by migrating valid data objects. The smaller the storage space that can be released by migrating valid data objects, the greater the cost required to migrate valid data objects to a new storage space to release holes, and the greater the hole compression cost. It can be seen that the size of the storage space that can be released by migrating valid data objects is inversely correlated with the cost required to migrate valid data objects to a new storage space to release holes. For example, the hole compression cost is inversely proportional to the size of the storage space that can be released by migrating valid data objects.
[0091] The distributed storage system may calculate the hole compression cost corresponding to the data aggregation file based only on the correlation between one parameter among the discrete degree of holes, the number of valid data objects, the valid data object rate, and the size of storage space that can be released by migrating valid data objects, and the cost required to migrate valid data objects to new storage space to release holes. It may also calculate the hole compression cost corresponding to the data aggregation file based on the correlation between multiple or all parameters among the discrete degree of holes, the number of valid data objects, the valid data object rate, and the size of storage space that can be released by migrating valid data objects, and the cost required to migrate valid data objects to new storage space to release holes.
[0092] For example, the hole compression cost is C, and the hole discreteness is I 离散 , the number of valid data objects is N 有效 , the total number of objects is N 总 , that is, the effective data object rate is N 有效 / N 总 , the size of the storage space that can be released by migrating valid data objects is S 释放 .
[0093] If the data object parameters only include the discrete degree of the holes, that is, the distributed storage system only calculates the hole compression cost corresponding to the data aggregation file based on the positive correlation between the discrete degree of the holes and the cost of migrating valid data objects to new storage space to release the holes. For example, C=I 离散 、C=k1xI 离散 、C=a1xI 离散 +b1, etc., which are all reasonable, wherein k1, a1 and b1 are constants preset based on experience.
[0094] If the data object parameter only includes the number of valid data objects, that is, the distributed storage system only calculates the hole compression cost corresponding to the data aggregation file based on the positive correlation between the number of valid data objects and the cost of migrating the valid data objects to the new storage space to release the hole, for example, C=N 有效 、C=k2xN 有效 、C=a2xN 有效 +b2, etc., which are all reasonable, where k2, a2 and b2 are constants preset based on experience.
[0095] If the data object parameters only include the valid data object rate, that is, the distributed storage system only calculates the hole compression cost corresponding to the data aggregation file based on the positive correlation between the valid data object rate and the cost required to migrate the valid data objects to the new storage space to release the hole, for example, C=N 有效 / N总 、C=k3x N 有效 / N 总 、C=a3x N 有效 / N 总 +b3, etc., which are all reasonable, wherein k3, a3 and b3 are constants preset based on experience.
[0096] If the data object parameters only include the size of the storage space that can be released by migrating valid data objects, that is, the distributed storage system only calculates the hole compression cost corresponding to the data aggregation file based on the inverse correlation between the size of the storage space that can be released by migrating valid data objects and the cost required to migrate valid data objects to new storage space to release holes, for example, C=1 / S 释放 、C=k4x1 / S 释放 、C=a4x1 / S 释放 +b4, etc., which are all reasonable, wherein k4, a4 and b4 are constants preset based on experience.
[0097] If the data object parameters include the discreteness of the holes and the number of valid data objects, then C = I 离散 xN 有效 、C=k5xI 离散 xN 有效 、C=a5xI 离散 xN 有效 +b5, etc. If the data object parameters include the discrete degree of the hole, the number of valid data objects, and the valid data object rate, then C = I 离散 xN 有效 xN 有效 / N 总 = I 离散 xN 2 有效 / N 总 、C=k6xI 离散 xN 2 有效 / N 总 、C=a6xI 离散 xN 2 有效 / N 总 +b6, etc.
[0098] If the data object parameters include the discrete degree of the holes, the number of valid data objects, and the size of the storage space that can be released by migrating valid data objects, then C = I 离散 xN 有效 / S 释放 、C=k7xI 离散 xN 有效 / S 释放、C=a7xI 离散 xN 有效 / S 释放 +b7, etc. If the data object parameters include the discrete degree of the hole, the number of valid data objects, the valid data object rate, and the size of the storage space that can be released by migrating valid data objects, then C=I 离散 xN 2 有效 / (N 总 xS 释放 )、C=k8xI 离散 xN 2 有效 / (N 总 xS 释放 )、C=a8xI 离散 xN 2 有效 / (N 总 xS 释放 )+b8. Among them, k5 to k8, a5 to a8, and b5 to b8 are constants preset based on experience.
[0099] It can be seen that in this embodiment, since the calculation of the hole compression cost takes into account one or more data object parameters including the discrete degree of holes in the data aggregation file, the number of valid data objects, the valid data object rate, and the size of the storage space that can be released by migrating valid data objects, and the positive correlation or negative correlation between the cost required to migrate valid data objects to new storage space to release holes, the calculated hole compression cost is more targeted, practical, and more accurate.
[0100] As an implementation manner of an embodiment of the present application, the step of calculating the hole compression cost corresponding to the data aggregation file based on the positive correlation or negative correlation between the data object parameter and the cost required to migrate the valid data object to the new storage space to release the hole may include:
[0101] In a case where the data object parameters include the discrete degree of the hole, the number of the valid data objects, the valid data object rate, and the size of the storage space that can be released by migrating the valid data objects, calculating a ratio between a product of a proportional parameter included in the data object parameters and an inverse parameter included in the data object parameters as a hole compression cost corresponding to the data aggregation file;
[0102] The proportional parameters include the discrete degree of the holes, the number of the valid data objects and the valid data object rate, and the inverse parameters include the size of the storage space that can be released by migrating the valid data objects.
[0103] Since the data object parameters may include multiple ones of the discrete degree of holes, the number of valid data objects, the valid data object rate, and the size of the storage space that can be released by migrating valid data objects, in one case, the data object parameters may include the discrete degree of holes, the number of valid data objects, the valid data object rate, and the size of the storage space that can be released by migrating valid data objects.
[0104] In one embodiment, when the data object parameters include the discrete degree of holes, the number of valid data objects, the valid data object rate, and the size of the storage space that can be released by migrating valid data objects, the distributed storage system can calculate the ratio between the product of the proportional parameters included in the data object parameters and the inverse parameters included in the data object parameters as the hole compression cost corresponding to the data aggregation file.
[0105] The proportional parameters include the discrete degree of holes, the number of valid data objects and the valid data object rate, and the inverse parameters include the size of the storage space that can be released by migrating valid data objects.
[0106] That is to say, the distributed storage system can use the product of data object parameters that are positively correlated with the cost of migrating valid data objects to new storage space to release holes, and the ratio of data object parameters that are negatively correlated with the cost of migrating valid data objects to new storage space to release holes as the hole compression cost, that is, the ratio of the product of the degree of discreteness, the number of valid data objects, and the valid data object rate to the size of the storage space that can be released by migrating valid data objects as the hole compression cost.
[0107] As an implementation method, the distributed storage system can calculate the hole compression cost according to the following formula: :
[0108] ;
[0109] in, is the discreteness of the void, is the number of valid data objects in the data aggregation file. The total number of objects included in the data aggregation file. is the effective data object rate, The amount of storage space that can be freed to migrate valid data objects.
[0110] It can be seen that in this embodiment, when calculating the hole compression cost, the distributed storage system takes into account multiple data object parameters such as the discrete degree of holes in the data aggregation file, the number of valid data objects, the valid data object rate, and the size of the storage space that can be released by migrating valid data objects. This can make the calculated hole compression cost more targeted, practical, and accurate.
[0111] As an implementation method of the present application, Figure 4 As shown, the calculation method of the discrete degree of the above-mentioned voids may include:
[0112] S401, for each data aggregation file, continuously arranging the holes in the data aggregation file according to the order of corresponding storage positions to obtain arranged continuous holes;
[0113] S402, calculating the span between the first preset position in the continuous hole and the corresponding position of the second preset position in the data aggregation file as the discrete degree of the hole corresponding to the data aggregation file.
[0114] The degree of discreteness of the holes can measure the degree of deviation of each hole from its center. The degree of discreteness of the holes can be determined by the position of each hole in the data aggregation file. If the positions of each hole in the data aggregation file are far apart, the degree of discreteness of the holes is high. If the positions of each hole in the data aggregation file are relatively concentrated, the degree of discreteness of the holes is low. Therefore, in order to calculate the degree of discreteness of the holes, two representative holes can be selected, and the span between the corresponding positions of the two holes in the data aggregation file can be calculated as the degree of discreteness of the holes.
[0115] In one embodiment, for each data aggregation file, the holes in the data aggregation file can be arranged continuously in the order of the corresponding storage positions to obtain the arranged continuous holes, and then the span between the first preset position in the continuous hole and the corresponding position of the second preset position in the data aggregation file is calculated as the discrete degree of the hole corresponding to the data aggregation file. The first preset position and the second preset position can be set according to actual needs, for example, the first preset position is the 1 / 4 position in the continuous hole, the second preset position is the 3 / 4 position in the continuous hole, or the first preset position is the 1 / 5 position in the continuous hole, and the second preset position is the 4 / 5 position in the continuous hole, which are all reasonable.
[0116] For example, Figure 5 As shown, the data aggregation file includes M valid data objects, namely valid data object 1 to valid data object M, and N holes, namely hole 1 to hole N. The offset of hole 1 is offset1, which is recorded as , the offset of hole N is offsetN, recorded as The distributed file system can continuously arrange the holes in the data aggregation file in the order of the corresponding storage positions to obtain arranged continuous holes 501, that is, to obtain arranged holes 1 to holes N. Then calculate the span between the corresponding positions of the 1 / 4 position and the 3 / 4 position in the continuous hole 501 in the data aggregation file.
[0117] Assume that the 1 / 4 position is located at the p-th byte of the j-th hole, corresponding to an offset of offsetJ in the data aggregation file, and the 3 / 4 position is located at the q-th byte of the k-th hole, corresponding to an offset of offsetK in the data aggregation file. Then the distributed file system can use the span between offsetK and offsetJ, i.e., offsetK-offsetJ, as the discrete degree of the hole corresponding to the data aggregation file.
[0118] It can be seen that in this embodiment, the holes in the data aggregation file can be arranged continuously in the order of the corresponding storage positions, and the span between the first preset position and the corresponding position of the second preset position in the aggregation file can be calculated as the discrete degree of the hole, which can more accurately reflect the degree of deviation of each hole from its center.
[0119] As an implementation of an embodiment of the present application, the step of calculating the span between the corresponding positions of the first preset position and the second preset position in the continuous hole in the data aggregation file as the discrete degree of the hole corresponding to the data aggregation file may include:
[0120] The continuous holes are normalized based on the span between the first hole and the last hole in the data aggregation file; and the span between the first preset position and the second preset position in the normalized continuous holes is determined as the discrete degree of the hole corresponding to the data aggregation file.
[0121] In order to make the calculated discreteness of the holes more accurate and unify the statistical distribution of the holes, the distributed storage system can normalize the calculated discrete holes. In one embodiment, the continuous holes can be normalized based on the span between the first hole and the last hole in the data aggregation file, and then the span between the first preset position and the second preset position in the normalized continuous hole is determined as the discreteness of the hole corresponding to the data aggregation file.
[0122] As an implementation method, the distributed storage system may normalize the calculated discrete holes based on the span from the first hole to the last hole in the data aggregation file according to the following formula:
[0123] ;
[0124] in, is the position corresponding to the kth hole in the data aggregation file when the second preset position is located at the qth byte of the kth hole, that is, the offset of the kth hole in the data aggregation file, is the position corresponding to the j-th hole in the data aggregation file when the first preset position is located at the p-th byte of the j-th hole, that is, the offset of the j-th hole in the data aggregation file, is the position of the last hole in the data aggregation file, that is, the offset of the last hole in the data aggregation file. is the position of the first hole in the data aggregation file, that is, the offset of the first hole in the data aggregation file. is the size of the last hole.
[0125] So, That is, the span from the last byte of the last hole to the first hole in the data aggregation file. That is, the span between the second preset position and the second preset position in the hole, which is divided by the span between the last byte of the last hole in the data aggregation file and the first hole, that is, the normalized discreteness of the hole based on the span between the first hole and the last hole in the data aggregation file.
[0126] In one implementation, the distributed storage system can calculate the span of each hole in the continuous hole according to the formula Calculate the first quartile And according to the formula Calculate the third quartile , and then determine the first preset position based on the first quartile, and determine the second preset position based on the third quartile.
[0127] Among them, the first quartile is the position corresponding to the hole at 25% of the continuous holes, which can represent the 1 / 4 position in the continuous holes after the holes in the data aggregation file are arranged continuously in the order of the corresponding storage positions; the third quartile is the position corresponding to the hole at 75% of the continuous holes, which can represent the 3 / 4 position in the continuous holes after the holes in the data aggregation file are arranged continuously in the order of the corresponding storage positions.
[0128] In this way, after obtaining the first quartile and the third quartile, the distributed storage system can use the position represented by the first quartile as the first preset position, and the position represented by the third quartile as the second preset position. For example, if the position represented by the first quartile is located at the pth byte of the jth hole in the continuous hole, then the pth byte of the jth hole is used as the first preset position, and the position represented by the third quartile is located at the qth byte of the kth hole in the continuous hole, then the qth byte of the kth hole is used as the second preset position.
[0129] It can be seen that in this embodiment, the distributed storage system can normalize the continuous holes and determine the span between the first preset position and the second preset position in the normalized continuous holes as the discrete degree of the holes corresponding to the data aggregation file. This can make the obtained discrete degree of the holes more accurate and unify the statistical distribution of the holes.
[0130] As an implementation of the embodiment of the present application, the calculation method of the size of the storage space that can be released by migrating the valid data objects may include:
[0131] The size of the storage space that can be released by migrating the valid data objects in the data aggregate file is calculated based on the number of data blocks and check blocks included in each valid data object in the data aggregate file and the size of the holes included in the data aggregate file.
[0132] EC Erasure Coding (EC) is a data protection method that is mainly used in network transmission and storage systems to improve data reliability and fault tolerance. The data is divided into multiple fragments (i.e., multiple data blocks), and check blocks are added for expansion and encoding, and then these data blocks are stored in different locations (such as disks, storage nodes, or other geographical locations). Specifically, a data object can be divided into n data blocks, and m check blocks are added, and then these data blocks are stored on n+m hard disks. In this way, the storage system can tolerate the failure of up to m hard disks. When no more than m hard disks fail, you only need to arbitrarily select n normal data blocks and check blocks to restore all the original data through calculation.
[0133] In the embodiment of the present application, the EC erasure code method is adopted, that is, when the distributed storage system stores the data object, the data object can be divided into multiple data blocks, and multiple check blocks are added. In one embodiment, for each data object, when the user uploads the data object, the distributed storage system can also add multiple redundant information for each data object. Among them, the redundant information can record information such as data verification, data recovery, and data location, which is not specifically limited here.
[0134] For example, Figure 6 As shown, the data aggregation file includes 5 data objects, namely objdata_1, objdata_2, objdata_3, objdata_4, and objdata_5. For the data object objdata_3, in addition to the data object, it also includes multiple redundant information, namely redundant information 1 to redundant information 5.
[0135] In the process of migrating valid data objects to new storage space in a distributed storage system, the size of the storage space that can be released by migrating the valid data objects in the data aggregate file is related to the number of data blocks, the number of check blocks included in each valid data object, and the size of the holes included in the data aggregate file. Therefore, the distributed storage system can calculate the size of the storage space that can be released by migrating the valid data objects in the data aggregate file based on the number of data blocks, the number of check blocks included in each valid data object in the data aggregate file, and the size of the holes included in the data aggregate file.
[0136] As an implementation method, the distributed storage system can calculate the size of the storage space that can be released by migrating the valid data objects in the data aggregation file according to the following formula: :
[0137] ;
[0138] in, The number of data blocks included in each valid data object in the data aggregation file, The number of check blocks included in each valid data object in the data aggregation file, The size of the holes included in the data aggregate file.
[0139] Then as an implementation method, the distributed storage system can calculate the hole compression cost according to the following formula: :
[0140] ;
[0141] in, is the discreteness of the void, is the number of valid data objects in the data aggregation file. The total number of objects included in the data aggregation file. This is the effective data object rate.
[0142] It can be seen that in this embodiment, since the number of data blocks, the number of check blocks included in each valid data object and the size of the holes included in the data aggregation file are taken into consideration when calculating the size of the storage space that can be released by migrating valid data objects, the calculated size of the storage space that can be released by migrating valid data objects is more accurate and has practical significance.
[0143] As an implementation of an embodiment of the present application, a data aggregation file has a corresponding index aggregation file, and the index aggregation file is used to store metadata of data objects included in the corresponding data aggregation file.
[0144] Since the data objects uploaded by users include not only the data objects themselves, but also the attribute information of the data objects, such as size, type, creation time, upload time, etc., which can be called metadata of the data objects, when users upload data objects, the distributed storage system aggregates and stores the metadata of the data objects in the index aggregation file corresponding to the data aggregation file while aggregating and storing the data objects in the data aggregation file. The position of the data object in the data aggregation file corresponds to the position of the metadata in the index aggregation file. For example, if data object A is arranged in the first position in the data aggregation file, then the metadata of data object A is also arranged in the first position in the index aggregation file.
[0145] Since the metadata is very small, two-level aggregation can be performed when storing the metadata corresponding to the data object, that is, the index aggregation file is first divided into multiple aggregation index areas, and then the metadata is stored in the corresponding aggregation index area. Each aggregation index area can store a preset number of metadata. In one embodiment, the distributed storage system can first store the metadata in the first aggregation index area, and when the number of metadata stored in the first aggregation index area reaches a preset number, continue to store it in the second aggregation index area, that is, store the metadata in each aggregation index area in turn. For each metadata, when the user uploads the data object, the distributed storage system can add multiple redundant information for each metadata, and can also add a magic number to mark the format of the file or protocol.
[0146] The distributed storage system can also add redundant information and verification information to each aggregate index area. The redundant information can record information such as data verification, data recovery, and data location, which is not specifically limited here. The verification information can ensure that the storage space where the metadata is written is free of abnormalities.
[0147] For example, Figure 7As shown, the index aggregation file is divided into 5 aggregation index areas, namely, aggregation index area 1 to aggregation index area 5, each aggregation index area can store 10 metadata, such as metadata entry 1 to metadata entry 10, redundant information 6 to redundant information 9 and verification information in aggregation index area 3, and redundant information 6 to redundant information 9 can record the location of metadata entry 1 to metadata entry 10. For metadata entry 2, in addition to including meta data, the distributed storage system can add a magic number Magic_num1 and multiple redundant information, such as redundant information 10 to redundant information 14, to record information such as data verification and data recovery.
[0148] The index aggregation file and the data aggregation file exist in pairs, and the total number of metadata in the index aggregation file is the same as the total number of objects included in the data aggregation file. For example, if the index aggregation file includes 5 aggregation index areas, each aggregation index area includes 10 metadata, then the index aggregation file includes 50 metadata, and the corresponding data aggregation file includes 50 objects.
[0149] The above method may further include:
[0150] The metadata included in the index aggregation file to be processed is written into a new index aggregation file, the index aggregation file to be processed is deleted, and a corresponding relationship between the new data aggregation file and the new index aggregation file is established, wherein the index aggregation file to be processed is an index aggregation file corresponding to the data aggregation file to be processed whose corresponding hole compression cost is not greater than a preset threshold.
[0151] In the case where a user deletes a data object, the metadata corresponding to the data object will also be deleted accordingly. Similarly, in the case where a data object expires, the metadata corresponding to the data object will also expire, so that in the index aggregation file, the storage space occupied by the metadata of the deleted data object, or the storage space occupied by the metadata of the expired data object will also form a hole accordingly. Then, when the distributed storage system writes the valid data objects in the pending data aggregation file whose corresponding hole compression cost is not greater than the preset threshold into the new data aggregation file, it should also write the index aggregation file corresponding to the pending data aggregation file whose corresponding hole compression cost is not greater than the preset threshold into the new index aggregation file.
[0152] In one embodiment, the distributed storage system writes the metadata included in the index aggregation file to be processed into a new index aggregation file, that is, the metadata is stored in the new index aggregation file, and then the index aggregation file to be processed can be deleted, and a corresponding relationship between the new data aggregation file and the new index aggregation file is established to ensure the corresponding relationship between the data object and the metadata of the data object, thereby releasing the holes in the index aggregation file to be processed, that is, there are no holes, and hole compression is achieved.
[0153] Among them, the index aggregation file to be processed is the index aggregation file corresponding to the data aggregation file to be processed whose corresponding hole compression cost is not greater than the preset threshold. The new index aggregation file refers to a blank file with continuous storage space and no holes. The correspondence between the new data aggregation file and the new index aggregation file can be a name correspondence, etc., which is not specifically limited here. For example, the name of the new data aggregation file is aaa.data, and the name of the new index aggregation file is aaa.index. In this way, the correspondence between the new data aggregation file and the new index aggregation file is established with the same name.
[0154] For example, there is one data aggregation file to be processed whose corresponding hole compression cost is not greater than the preset threshold, including 20 valid data objects and 10 holes. Correspondingly, the index aggregation file to be processed includes 20 metadata and 10 holes, the aggregation index area 1 includes 6 metadata and 4 holes, the aggregation index area 2 includes 8 metadata and 2 holes, and the aggregation index area 3 includes 6 metadata and 4 holes. Then the distributed storage system can write the 20 metadata included in the aggregation index area 1 to the aggregation index area 3 in the index aggregation file to be processed into a new index aggregation file, which only occupies the aggregation index area 1 and the aggregation index area 2 of the new index aggregation file.
[0155] Then, the distributed storage system can delete the to-be-processed index aggregation file and establish a corresponding relationship between a new data aggregation file and a new index aggregation file, that is, establish a corresponding relationship between a data object and metadata.
[0156] It can be seen that in this embodiment, the distributed storage system writes the metadata included in the index aggregation file to be processed into a new index aggregation file, deletes the index aggregation file to be processed, and establishes a corresponding relationship between the new data aggregation file and the new index aggregation file, so that the index aggregation files corresponding to the data aggregation files with smaller hole compression costs are aggregated, that is, the data aggregation files with a hole compression cost not greater than a preset threshold are aggregated, and the original data aggregation files are deleted after aggregation. In this way, data aggregation files with smaller hole compression costs are preferentially selected for hole compression, which can achieve the purpose of consuming the least resources and releasing as much storage space as possible as quickly as possible, thereby improving the storage space utilization rate.
[0157] As an implementation method of the present application, Figure 8 As shown, the above method may also include:
[0158] S801, obtaining a data object to be stored, and allocating a storage path for the data object to be stored;
[0159] The storage path is determined based on the name prefix of the data object to be stored.
[0160] Since the distributed storage system needs to store the data object to be stored in the data aggregation file when it obtains the data object to be stored, the distributed storage system can allocate a storage path for the data object to be stored, that is, determine the storage path of the data aggregation file to be written into the data object to be stored. The storage path is determined based on the name prefix of the data object to be stored, that is, the name prefix of the data object includes information of the storage path.
[0161] In one embodiment, when the distributed storage system obtains the data object to be stored, a storage path can be allocated to the data object to be stored according to the node, date and name prefix where the data object to be stored is stored. Specifically, when the distributed storage system obtains the data object to be stored, the node where the data object to be stored is to be stored can be determined according to load balancing, and then a folder including the name prefix under the node can be determined according to the name prefix, and a folder including the date can be further determined under the file including the name prefix according to the date, and then the data object to be stored can be stored in the data aggregation file under the folder including the date.
[0162] For example, the node where the data object to be stored is stored is node001, the name prefix is prefix001, and the date is 240628, then the storage path allocated by the distributed file system to the data object to be stored is / node001 / prefix001 / date240628.
[0163] S802, determining whether there is a first data aggregation file under the storage path whose expiration rule is the same as the prefix expiration rule represented by the name prefix of the data object to be stored; if so, executing step S803; if not, executing step S804;
[0164] S803, writing the data object to be stored into the first data aggregation file, and writing the metadata of the data object to be stored into the index aggregation file corresponding to the first data aggregation file;
[0165] Object storage supports configuring prefix expiration rules. Prefix expiration rules are to configure data objects with the same prefix to expire at the same time. For example, if you configure prefix 001 to expire in 2 days, then all data objects with prefix 001 will be deleted after 2 days.
[0166] After the distributed storage system allocates a storage path for the data object to be stored, it can be determined whether there is a first data aggregation file under the storage path that has an expiration rule that is the same as the prefix expiration rule represented by the name prefix of the data object to be stored. The data objects stored in the first data aggregation file are data objects that expire at the same time. In other words, the distributed storage system can determine whether there is already a data aggregation file under the storage path that stores data objects that expire at the same time as the data object to be stored.
[0167] If it exists, it means that before storing the data object to be stored, the user has uploaded a data object that expired at the same time, that is, the corresponding first data aggregation file already exists. Then the distributed storage system can write the data object to be stored to the first data aggregation file, and write the metadata of the data object to be stored to the index aggregation file corresponding to the first data aggregation file.
[0168] For example, under the path / node001 / prefix001 / date240628, there is a data aggregation file file001.data and an index aggregation file file001.index corresponding to the data aggregation file. The storage path allocated by the distributed storage system for the data object to be stored is / node001 / prefix001 / date240628. It can be determined that under the storage path, there is a first data aggregation file whose expiration rule is the same as the prefix expiration rule represented by the name prefix of the data object to be stored, that is, there is a data aggregation file file001.data whose expiration rule is prefix 001 and an index aggregation file file001.index corresponding to the data aggregation file file001.data. Then, the distributed storage system can write the data object to be stored into the data aggregation file file001.data, and write the metadata of the data object to be stored into the index aggregation file file001.index corresponding to the data aggregation file file001.data.
[0169] S804. Under the storage path, create a second data aggregation file and an index aggregation file corresponding to the second data aggregation file, write the data object to be stored into the second data aggregation file, and write the metadata of the data object to be stored into the index aggregation file corresponding to the second data aggregation file, wherein the expiration rule of the second data aggregation file is the prefix expiration rule represented by the name prefix of the data object to be stored.
[0170] If it does not exist, it means that before storing the data object to be stored, no user has uploaded a data object that expired at the same time, that is, the distributed storage system stores the data object with the expiration date for the first time. In this case, the distributed storage system can create a second data aggregation file and an index aggregation file corresponding to the second data aggregation file under the storage path, and write the data object to be stored to the second data aggregation file, and write the metadata of the data object to be stored to the index aggregation file corresponding to the second data aggregation file, wherein the expiration rule of the second data aggregation file is the prefix expiration rule represented by the name prefix of the data object to be stored, that is, the expiration time of the data object stored in the second data aggregation file is the same as that of the data object to be stored.
[0171] For example, the storage path allocated by the distributed storage system to the data object to be stored is / node001 / prefix001 / date240628. It can be determined that there is no data aggregation file under the storage path whose expiration rule is the same as the prefix expiration rule represented by the name prefix of the data object to be stored, that is, there is no data aggregation file file001.data with an expiration rule of prefix 001 and an index aggregation file file001.index corresponding to the data aggregation file file001.data. Then the distributed storage system can create the data aggregation file file001.data and the index aggregation file file001.index corresponding to the data aggregation file file001.data, and write the data object to be stored to the data aggregation file file001.data, and write the metadata of the data object to be stored to the index aggregation file file001.index corresponding to the data aggregation file file001.data.
[0172] In one embodiment, the distributed storage system can write the metadata of the data object to be stored into the index aggregation file corresponding to the data aggregation file, and at the same time, write the metadata into the distributed database. In this way, when a user searches for a data object, the metadata of the data object can be quickly and conveniently found from the distributed database, and then the corresponding data object can be found from the data aggregation file.
[0173] For example, Fig. 9As shown, the distributed storage system uses a distributed file system (DFS) as a storage engine. The distributed storage system may include a distributed file system, an object gateway, an object SDK (Software Development Kit), etc. When a client user sends a request instruction to upload a data object to the distributed object gateway, the object gateway sends an instruction to initialize the upload interface to the object SDK. The object SDK can initialize internal parameters and assign a storage path to the data object, that is, determine the storage directory. The object SDK can determine the storage directory based on the node assigned to the data object, the prefix expiration rule, and the write date. If a storage directory already exists, there is no need to create it. The data object can be directly written to the data aggregation file under the storage directory, and the metadata of the data object can be written to the index aggregation file corresponding to the data aggregation file.
[0174] If the storage directory does not exist, the object SDK sends a command to create the directory to DFS, and returns success to the object SDK after DFS successfully creates the directory. The object SDK sends a command to create a data aggregation file and the index aggregation file corresponding to the data aggregation file to DFS. If DFS is successfully created, it returns success to the object SDK. The object SDK returns success to the object gateway, and the object gateway uploads the data object to the object SDK. When the object SDK receives the request command to upload the data object, it sends a command to open the newly created data aggregation file and the index aggregation file corresponding to the newly created data aggregation file to DFS. After opening, DFS returns success to the object SDK.
[0175] The object sdk sends instructions for writing the data object and writing the metadata of the data object to DFS respectively. After receiving the instructions, DFS can write the data object to the newly created data aggregation file, return success to the object sdk, and write the metadata of the data object to the index aggregation file corresponding to the newly created data aggregation file, and return success to the object sdk.
[0176] The object SDK can also send instructions for inserting metadata of data objects to the distributed database. After storing the metadata, the distributed database returns success to the object SDK. The object SDK returns success to the object gateway. The object gateway returns success to the client, informing the user that the uploaded data object has been stored.
[0177] Furthermore, the object SDK can send an instruction to DFS to record the number of valid data objects and the total number of objects in the data aggregation file in the extended attributes of the index aggregation file corresponding to the data aggregation file. DFS records the number of valid data objects and the total number of objects in the data aggregation file in the extended attributes of the index aggregation file corresponding to the data aggregation file, that is, updates the index aggregation file extended attributes and returns success. The object SDK can send an instruction to DFS to close the data aggregation file and the index aggregation file corresponding to the data aggregation file. After DFS closes the data aggregation file and the index aggregation file corresponding to the data aggregation file, it returns success to the object SDK to complete the entire process of data object upload.
[0178] It can be seen that in this embodiment, since the distributed storage system can determine the data aggregation file into which the data object to be stored is written according to the prefix expiration rule, and write multiple data objects that expire on the same day into the same data aggregation file, the data objects in the data aggregation file will all expire on the same day. In the event of expiration, the entire data aggregation file can be directly deleted, and no holes will be generated due to the expiration of a certain data object. This will significantly reduce the generation of holes.
[0179] As an implementation method of an embodiment of the present application, a distributed storage system includes at least one object bucket, and the storage pool corresponding to each object bucket includes at least one data aggregation file. Each object bucket has an expiration rule list, and the expiration rule list includes an expiration rule identifier corresponding to the data aggregation file included in the storage pool corresponding to the object bucket.
[0180] An object bucket is a container for storing data objects in object storage. Object storage provides a flat storage method based on object buckets and data objects. All data objects in an object bucket are at the same logical level, eliminating the multi-level tree directory structure in the file system. Each object bucket has its own storage category, access rights, region, and other attributes. Users can create object buckets with different storage categories and access rights in different regions, and configure more advanced attributes to meet the storage requirements of different scenarios.
[0181] Each object bucket corresponds to a storage pool, which can include multiple nodes. Multiple folders can be created under each node based on expiration rules, and the folder names can be named based on the expiration rule identifiers. For example, the expiration rule list includes rules for 2-day expiration, 3-day expiration, and 5-day expiration, and the expiration rule identifiers are prefix 001, prefix 002, and prefix 003, respectively. Then, 3 folders can be created under each node, and the folder names include prefix 001, prefix 002, and prefix 003, respectively.
[0182] The method for determining the above-mentioned data aggregation file to be processed may include:
[0183] Traverse the expiration rule list included in the object bucket, and each time an expiration rule identifier is traversed, search for the corresponding data aggregation file included in the storage pool corresponding to the object bucket, whose expiration rule is the same as the expiration rule identified by the currently traversed expiration rule identifier, as the data aggregation file to be processed.
[0184] In order to obtain the data aggregation file to be processed, the distributed storage system can determine an expiration rule identifier from the expiration rule list included in the object bucket, and then obtain the corresponding data aggregation file whose expiration rule is the same as the expiration rule identified by the expiration rule identifier from the data aggregation file included in the storage pool corresponding to the object bucket as the data aggregation file to be processed.
[0185] In one embodiment, the distributed storage system can traverse the expiration rule list included in the object bucket. Every time an expiration rule identifier is traversed, the corresponding data aggregation file included in the storage pool corresponding to the object bucket is searched for the expiration rule that is the same as the expiration rule identified by the currently traversed expiration rule identifier, as the data aggregation file to be processed.
[0186] For example, the expiration rule list includes 5 expiration rules, namely, 2-day expiration, 3-day expiration, 5-day expiration, 7-day expiration, and 10-day expiration, and the expiration rule identifiers are prefix 001, prefix 002, prefix 003, prefix 004, and prefix 005. When the distributed storage system traverses to the expiration rule identifier as prefix 002, it can search for a data aggregation file with an expiration rule of 3 days from the storage pool corresponding to the object bucket as the data aggregation file to be processed.
[0187] Of course, if the storage pool corresponding to the object bucket includes multiple layers of folders, such as folders created based on nodes / folders created based on expiration rules / folders created based on dates, then after determining the expiration rule identifier, the distributed storage system can obtain the data aggregation file from the folder including the date corresponding to the expiration rule identifier as the data aggregation file to be processed.
[0188] For example, Fig.10 As shown, the storage pool corresponding to the object bucket includes three nodes, namely node001, node002, and node003. Under each node, there are multiple folders created based on expiration rules. For example, under node node002, there are folders created based on the rules of 2-day expiration, 3-day expiration, and 5-day expiration, namely folders of prefix 001, prefix 002, and prefix 003. Under the folders created based on the expiration rules, folders can be created based on dates. For example, under the folder of prefix 002, multiple folders are created based on dates, namely folders of write date 001, write date 002, and write date 003.
[0189] In the folder created based on the date, create data aggregation files corresponding to the expiration rules and index aggregation files corresponding to the data aggregation files. For example, create data aggregation files file001.data, file002.data, file003.data, and index aggregation files file001.index corresponding to file001.data, file002.index corresponding to file002.data, and file003.index corresponding to file003.data in the folder with the write date 002. Then, after determining that the expiration rule identifier is prefix 002, the distributed storage system can obtain data aggregation files file001.data, file002.data, and file003.data from the folder including the expiration rule identifier prefix 002 and the date of the write date 002 as the data aggregation files to be processed.
[0190] It can be seen that in this embodiment, the distributed storage system can traverse the expiration rule list included in the object bucket. Every time an expiration rule identifier is traversed, the corresponding data aggregation file included in the storage pool corresponding to the object bucket is searched for the expiration rule that is the same as the expiration rule identified by the currently traversed expiration rule identifier, as the data aggregation file to be processed. In this way, data aggregation files that meet the same expiration rule and expire on the same day can be processed, so as to minimize the generation of holes while selecting data aggregation files with a smaller hole compression cost for hole compression, thereby improving storage space utilization.
[0191] As an implementation of an embodiment of the present application, after writing the data object to be stored into the target data aggregation file, wherein the target data aggregation file is the first data aggregation file or the second data aggregation file, the above method may further include:
[0192] Determine the number of valid data objects and the total number of objects in the target data aggregation file; and record the number of valid data objects and the total number of objects in an extended attribute of an index aggregation file corresponding to the target data aggregation file.
[0193] After the distributed storage system writes the data object to be stored into the first data aggregation file or writes the data object to be stored into the second data aggregation file, the number of valid data objects and the total number of objects included in the first data aggregation file or the second data aggregation file change. In order to facilitate the distributed storage system to calculate the void compression cost, after writing the data object to be stored into the target data aggregation file, the number of valid data objects and the total number of objects in the target data aggregation file can be determined, and then the number of valid data objects and the total number of objects can be recorded in the extended attributes of the index aggregation file corresponding to the target data aggregation file.
[0194] It can be seen that in this embodiment, the distributed storage system can determine the number of valid data objects and the total number of objects in the target data aggregation file, and record the number of valid data objects and the total number of objects in the extended attributes of the index aggregation file corresponding to the target data aggregation file, so as to facilitate the distributed storage system to calculate the hole compression cost in a timely and accurate manner.
[0195] As an implementation of an embodiment of the present application, the above-mentioned data aggregation file has a corresponding index aggregation file, and the index aggregation file is used to store metadata of the data objects included in the corresponding data aggregation file.
[0196] like Fig.11 As shown, the above method may also include:
[0197] S1101, obtaining a data object deletion instruction, and determining an index aggregation file corresponding to the data object to be deleted indicated by the data object deletion instruction.
[0198] When a user deletes a data object, the distributed storage system may obtain a data object deletion instruction, and may determine an index aggregation file corresponding to the data object to be deleted indicated by the data object deletion instruction.
[0199] In one embodiment, if the distributed storage system writes the metadata of the data object to be stored into the index aggregation file corresponding to the data aggregation file and writes the metadata into the distributed database, then when the data object is deleted, the metadata of the data object can be first found from the distributed database, and then the corresponding data object can be found from the data aggregation file.
[0200] Then, if the distributed storage system can obtain the data object deletion instruction, it can search the distributed database for the metadata corresponding to the data object to be deleted indicated by the data object deletion instruction, and then determine the corresponding index aggregation file based on the found metadata. For example, if the user deletes the data object objdata_1 stored in the data aggregation file file001.data, and if the distributed storage system obtains the data object deletion instruction, it can search the distributed storage system for the metadata corresponding to the data object objdata_1, and then determine the index aggregation file corresponding to the metadata, which is file001.index.
[0201] S1102: Set the deletion mark corresponding to the metadata of the to-be-deleted data object in the determined index aggregation file to a preset value.
[0202] The metadata corresponding to the deletion mark being the preset value and the data object corresponding to the metadata in the data aggregation file are invalid data.
[0203] In one embodiment, after the distributed storage system determines the index aggregation file corresponding to the data object to be deleted, the deletion mark corresponding to the metadata of the data object to be deleted in the determined index aggregation file can be set to a preset value. The metadata corresponding to the deletion mark is the preset value and the data object corresponding to the metadata in the data aggregation file is invalid data. The preset value can be a number or a letter, which is not specifically limited here.
[0204] That is to say, when deleting a data object, there is no need to delete the data object from the data aggregation file and the metadata corresponding to the data object from the index aggregation file. Instead, the metadata corresponding to the deleted data object is marked with a preset value, so that it can be identified that the storage space occupied by the data object and the metadata corresponding to the data object are both empty.
[0205] For example, after the distributed storage system determines the index aggregation file corresponding to the data object to be deleted, the deletion mark corresponding to the metadata of the data object to be deleted in the determined index aggregation file can be set to 1, so that the metadata and the data object corresponding to the metadata in the data aggregation file are both invalid data.
[0206] S1103: Count the total number of data objects in the data aggregation file corresponding to the determined index aggregation file.
[0207] S1104: Determine the number of valid data objects in the data aggregation file corresponding to the index aggregation file according to the number of deletion marks with a preset value in the determined index aggregation file.
[0208] After the distributed storage system sets the deletion mark corresponding to the metadata of the data object to be deleted to a preset value, it indicates that the data object to be deleted is deleted. Then, the total number of objects in the data aggregation file corresponding to the determined index aggregation file can be counted, and the number of valid data objects in the data aggregation file corresponding to the index aggregation file can be further determined based on the number of deletion marks with preset values in the determined index aggregation file.
[0209] For example, the total number of objects in the data aggregation file corresponding to the data object to be deleted is N, including N1 valid objects and N-N1 holes. Then, in the case of deleting the data object to be deleted, the distributed storage system counts the total number of objects in the data aggregation file corresponding to the index aggregation file determined to be N, and according to the number of deletion marks with preset values in the determined index aggregation file being N-N1, it can be determined that the number of valid data objects in the data aggregation file corresponding to the index aggregation file is N1.
[0210] S1105 , based on the total number and the number of valid data objects in the data aggregation file corresponding to the index aggregation file, calculate the hole compression cost corresponding to the data aggregation file corresponding to the index aggregation file.
[0211] S1106, recording the total number, the number of valid data objects in the data aggregation file corresponding to the index aggregation file, and the hole compression cost corresponding to the data aggregation file corresponding to the index aggregation file into the extended attributes of the index aggregation file.
[0212] After determining the total number and the number of valid data objects in the data aggregation file corresponding to the index aggregation file, the hole compression cost corresponding to the data aggregation file corresponding to the index aggregation file can be calculated. The specific implementation method of calculating the hole compression cost is the same as the method of calculating the hole compression cost in the above steps, which will not be repeated here.
[0213] The distributed storage system may record the total number, the number of valid data objects in the data aggregation file corresponding to the index aggregation file, and the hole compression cost corresponding to the data aggregation file corresponding to the index aggregation file in the extended attributes of the index aggregation file.
[0214] For example, Fig.12As shown, the distributed storage system uses a distributed file system (DFS) as a storage engine. The distributed file system may include a distributed file system, an object gateway, an object SDK, etc. When a client user sends a request instruction to delete a data object to the distributed object gateway, the object gateway sends an instruction to initialize the deletion interface to the object SDK, and the object SDK sends an instruction to query the metadata of the data object to the distributed database. When the distributed database finds the metadata of the data object, it returns a success message to the object SDK. When the object SDK determines that the metadata of the data object has been found, it may create an asynchronous task to set the deletion mark corresponding to the metadata in the index aggregation file to 1.
[0215] The object SDK sends an instruction to delete the metadata of the data object to the distributed database. The distributed database first deletes the metadata of the data object and returns success to the object SDK. The object SDK returns success to the object gateway, and then the object gateway returns success to the client, informing the user that the data object has been deleted.
[0216] The object SDK sends an instruction to open the index aggregation file corresponding to the data aggregation file to DFS. After opening the index aggregation file corresponding to the data aggregation file, DFS returns success to the object SDK. The object SDK sends an instruction to update the deletion mark corresponding to the metadata of the deleted data object in the index aggregation file to 1 to DFS. After DFS sets the deletion mark corresponding to the metadata of the deleted data object in the index aggregation file to 1, it returns success to the object SDK. The object SDK can calculate the hole compression cost based on the total number of objects in the data aggregation file and the number of valid data objects in the data aggregation file, and then send an instruction to update the extended attributes of the index aggregation file to DFS. After recording the total number of objects in the data aggregation file, the number of valid data objects in the data aggregation file, and the hole compression cost corresponding to the data aggregation file to the extended attributes of the index aggregation file, DFS returns success to the object SDK to complete the entire process of data object deletion.
[0217] It can be seen that in this embodiment, after the distributed storage system deletes the data object to be deleted, the total number of objects in the data aggregation file corresponding to the determined index aggregation file can be counted, the number of valid data objects in the data aggregation file corresponding to the index aggregation file can be determined, and the hole compression cost corresponding to the data aggregation file corresponding to the index aggregation file can be calculated, and the total number, the number of valid data objects in the data aggregation file corresponding to the index aggregation file, and the hole compression cost corresponding to the data aggregation file corresponding to the index aggregation file can be recorded in the extended attributes of the index aggregation file. In this way, each time a data object is deleted, the total number, the number of valid data, and the hole compression cost can be re-recorded, and the hole compression cost can be quickly and conveniently obtained from the index aggregation file.
[0218] As an implementation of an embodiment of the present application, the step of obtaining the hole compression cost corresponding to the to-be-processed data aggregate file may include:
[0219] The recorded hole compression cost is obtained from the extended attribute of the index aggregation file corresponding to the data aggregation file to be processed.
[0220] Since the distributed storage system calculates the hole compression cost and records it in the extended attribute of the index aggregation file, the distributed storage system can obtain the recorded hole compression cost from the extended attribute of the index aggregation file corresponding to the data aggregation file to be processed.
[0221] It can be seen that in this embodiment, the distributed storage system obtains the recorded hole compression cost from the extended attribute of the index aggregation file corresponding to the data aggregation file to be processed, so that the hole compression cost can be obtained quickly and accurately.
[0222] Fig.13 A specific flow chart of the data aggregation method provided in the embodiment of the present application. Fig.13 The data aggregation method provided in the embodiment of the present application is introduced by way of example. Fig.13 As shown, the data aggregation method provided in the embodiment of the present application may include the following steps:
[0223] S1301, traverse all object buckets;
[0224] A distributed storage system may include multiple object buckets. In order to perform hole compression on data aggregate files with low hole compression cost, the distributed storage system may traverse all object buckets to implement hole compression for the data aggregate files included in the storage pool corresponding to each object bucket.
[0225] S1302, for each traversed object bucket, query the expiration rule list of the object bucket;
[0226] Each object bucket has an expiration rule list. For each traversed object bucket, the distributed storage system can query the expiration rule list of the object bucket. The expiration rule list includes the expiration rule identifier corresponding to the data aggregation file included in the storage pool corresponding to the object bucket. For example, the expiration rule list includes 5 expiration rule identifiers, namely prefix 001, prefix 002, prefix 003, prefix 004 and prefix 005, which respectively indicate 2 days expiration, 3 days expiration, 5 days expiration, 7 days expiration and 10 days expiration.
[0227] S1303, traverse each expired rule identifier in the expired rule list;
[0228] S1304, for each traversed expiration rule identifier, searching for a date list including the expiration rule identifier;
[0229] The distributed storage system traverses each expiration rule identifier in the expiration rule list, and for each traversed expiration rule identifier, searches for a date list under a folder including the expiration rule identifier. Since the storage pool corresponding to the traversed object bucket may include multiple nodes, the distributed storage system may search for a date list under a folder including the expiration rule identifier from any node. For example, when the distributed storage system traverses to the expiration rule identifier prefix 003, for node 001, a date list under a folder including the expiration rule identifier may be searched from node 001, such as searching for a date list under node 001 / prefix 003, and the date list includes write date 001, write date 002, and write date 003.
[0230] S1305, traverse the files in the date list;
[0231] The distributed storage system can traverse the files under each date list, such as traversing the files under writing date 001, writing date 002, and writing date 003.
[0232] S1306, for each traversed date list, query the extended attribute of each data aggregation file in the date list;
[0233] S1307, obtaining the hole compression cost recorded in each extended attribute;
[0234] For each traversed date list, the distributed storage system can query the extended attributes of the index aggregation file corresponding to each data aggregation file under the date list to obtain the hole compression cost of each data aggregation file. For example, the write date 002 includes data aggregation files file001.data, file002.data, file003.data, and the index aggregation file file001.index corresponding to file001.data, the index aggregation file file002.index corresponding to file002.data, and the index aggregation file file003.index corresponding to file003.data. Then the distributed storage system can query the extended attributes of the index aggregation files file001.index, file002.index, and file003.index, and obtain the hole compression cost of each data aggregation file from the extended attributes.
[0235] S1308, determining whether the compression cost of each hole is less than a preset threshold;
[0236] S1309: Aggregate the data aggregation files whose corresponding hole compression costs are not greater than a preset threshold.
[0237] The distributed storage system can determine whether each hole compression cost is less than a preset threshold, and then aggregate the data aggregation files whose corresponding hole compression costs are not greater than the preset threshold, that is, write the valid data objects in the data aggregation files whose corresponding hole compression costs are not greater than the preset threshold into new data aggregation files, and delete the data aggregation files whose corresponding hole compression costs are not greater than the preset threshold.
[0238] It can be seen that in the scheme provided in the embodiment of the present application, it is applied to a distributed storage system, the distributed storage system includes at least one data aggregation file, the data aggregation file is used to aggregate and store multiple data objects, and the distributed storage system can obtain the hole compression cost corresponding to the data aggregation file to be processed, wherein the data aggregation file to be processed includes valid data objects and holes, and the hole compression cost is used to represent the cost required to migrate the valid data objects to the new storage space to release the holes, write the valid data objects in the data aggregation file to be processed whose corresponding hole compression cost is not greater than the preset threshold into the new data aggregation file, and delete the data aggregation file to be processed whose corresponding hole compression cost is not greater than the preset threshold. Since the hole compression cost can represent the cost required to migrate the valid data objects to the new storage space to release the holes, then when the hole compression cost corresponding to the data aggregation file to be processed is obtained, the data aggregation files with smaller hole compression costs can be aggregated, that is, the data aggregation files with hole compression costs not greater than the preset threshold are aggregated, and the original data aggregation files are deleted after aggregation, so that the data aggregation files with smaller hole compression costs are preferentially selected for hole compression, which can achieve the purpose of consuming the least resources and releasing as much storage space as possible as quickly as possible, thereby improving the storage space utilization rate.
[0239] Corresponding to the above data aggregation method, the embodiment of the present application further provides a data aggregation device. The following is an introduction to a data aggregation device provided by the embodiment of the present application.
[0240] like Fig.14 As shown, a data aggregation device is applied to a distributed storage system, the distributed storage system includes at least one data aggregation file, the data aggregation file is used to aggregate and store multiple data objects, the device includes:
[0241] The hole compression cost acquisition module 1410 is used to acquire the hole compression cost corresponding to the data aggregate file to be processed, wherein the data aggregate file to be processed includes valid data objects and holes, and the hole compression cost is used to represent the cost required to migrate the valid data objects to a new storage space to release the holes;
[0242] The valid data object writing module 1420 is used to write the valid data objects in the to-be-processed data aggregation file whose corresponding hole compression cost is not greater than a preset threshold into a new data aggregation file;
[0243] The file deletion module 1430 is used to delete the to-be-processed data aggregation file whose corresponding hole compression cost is not greater than a preset threshold.
[0244] It can be seen that in the scheme provided in the embodiment of the present application, it is applied to a distributed storage system, the distributed storage system includes at least one data aggregation file, the data aggregation file is used to aggregate and store multiple data objects, and the distributed storage system can obtain the hole compression cost corresponding to the data aggregation file to be processed, wherein the data aggregation file to be processed includes valid data objects and holes, and the hole compression cost is used to represent the cost required to migrate the valid data objects to the new storage space to release the holes, write the valid data objects in the data aggregation file to be processed whose corresponding hole compression cost is not greater than the preset threshold into the new data aggregation file, and delete the data aggregation file to be processed whose corresponding hole compression cost is not greater than the preset threshold. Since the hole compression cost can represent the cost required to migrate the valid data objects to the new storage space to release the holes, then when the hole compression cost corresponding to the data aggregation file to be processed is obtained, the data aggregation files with smaller hole compression costs can be aggregated, that is, the data aggregation files with hole compression costs not greater than the preset threshold are aggregated, and the original data aggregation files are deleted after aggregation, so that the data aggregation files with smaller hole compression costs are preferentially selected for hole compression, which can achieve the purpose of consuming the least resources and releasing as much storage space as possible as quickly as possible, thereby improving the storage space utilization rate.
[0245] As an implementation of the embodiment of the present application, the above-mentioned device may further include a hole compression cost calculation module, and the hole compression cost calculation module is used to calculate the hole compression cost, which may include:
[0246] A hole compression cost calculation submodule is used to calculate the hole compression cost for each data aggregation file according to the relationship between the data object parameters in the data aggregation file and the cost required to migrate the valid data object to the new storage space to release the hole;
[0247] Among them, the data object parameters include at least one of the discrete degree of holes, the number of valid data objects, the valid data object rate and the size of the storage space that can be released by migrating the valid data objects; the valid data object rate is the ratio of the number of valid data objects to the total number of objects included in the data aggregation file, and the total number of objects includes the number of valid data objects and the number of holes.
[0248] As an implementation of the embodiment of the present application, the above-mentioned device may further include a discrete degree calculation module, and the discrete degree calculation module is used to calculate the discrete degree of the hole, which may include:
[0249] A continuous hole acquisition submodule is used to continuously arrange the holes in each data aggregation file in the order of corresponding storage positions to obtain arranged continuous holes;
[0250] The discrete degree calculation submodule is used to calculate the span between the first preset position and the second preset position in the continuous hole corresponding to the position in the data aggregation file as the discrete degree of the hole corresponding to the data aggregation file.
[0251] As an implementation of the embodiment of the present application, the above-mentioned discrete degree calculation submodule may include:
[0252] A normalization processing unit, configured to normalize the continuous holes based on the span from the first hole to the last hole in the data aggregation file;
[0253] The discrete degree calculation unit is used to determine the span between the first preset position and the second preset position in the normalized continuous hole as the discrete degree of the hole corresponding to the data aggregation file.
[0254] As an implementation of an embodiment of the present application, the above-mentioned hole compression cost calculation submodule may include:
[0255] A hole compression cost calculation unit, configured to calculate a hole compression cost corresponding to the data aggregation file based on a positive correlation or an inverse correlation between a data object parameter and a cost required to migrate the valid data object to a new storage space to release a hole;
[0256] Among them, the discreteness of the holes, the number of valid data objects and the valid data object rate are positively correlated with the cost of migrating the valid data objects to new storage space to release the holes, and the size of the storage space that can be released by migrating the valid data objects is negatively correlated with the cost of migrating the valid data objects to new storage space to release the holes.
[0257] As an implementation of the embodiment of the present application, the above-mentioned hole compression cost calculation unit may include:
[0258] a hole compression cost calculation subunit, configured to calculate, when the data object parameters include the discrete degree of the hole, the number of the valid data objects, the valid data object rate, and the size of the storage space that can be released by migrating the valid data objects, a ratio between the product of the proportional parameter included in the data object parameters and the inverse parameter included in the data object parameters as the hole compression cost corresponding to the data aggregation file;
[0259] The proportional parameters include the discrete degree of the holes, the number of the valid data objects and the valid data object rate, and the inverse parameters include the size of the storage space that can be released by migrating the valid data objects.
[0260] As an implementation manner of the embodiment of the present application, the data aggregation file has a corresponding index aggregation file, and the index aggregation file is used to store metadata of the data objects included in the corresponding data aggregation file;
[0261] The above device may also include:
[0262] An index aggregation file determination module, used to obtain a data object deletion instruction, and determine an index aggregation file corresponding to the data object to be deleted indicated by the data object deletion instruction;
[0263] A preset value marking module, used to set the deletion mark corresponding to the metadata of the data object to be deleted in the determined index aggregation file to a preset value, wherein the metadata corresponding to the deletion mark is the preset value and the data object corresponding to the metadata in the data aggregation file is invalid data;
[0264] A total number statistics module is used to count the total number of objects in the data aggregation file corresponding to the determined index aggregation file;
[0265] A valid data object quantity determination module, used to determine the number of valid data objects in the data aggregation file corresponding to the index aggregation file according to the number of deletion marks with preset values in the determined index aggregation file;
[0266] A second hole compression cost calculation module, used to calculate the hole compression cost corresponding to the data aggregation file corresponding to the index aggregation file based on the total number and the number of valid data objects in the data aggregation file corresponding to the index aggregation file;
[0267] The second recording module is used to record the total number, the number of valid data objects in the data aggregation file corresponding to the index aggregation file, and the hole compression cost corresponding to the data aggregation file corresponding to the index aggregation file into the extended attributes of the index aggregation file.
[0268] The present application also provides a distributed storage system. Fig.15 As shown, including:
[0269] Memory 1501, used for storing computer programs;
[0270] The processor 1502 is used to implement the data aggregation method steps described in any of the above embodiments when executing the program stored in the memory 1501.
[0271] In addition, the above-mentioned distributed storage system may further include a communication bus and / or a communication interface, and the processor 1502, the communication interface, and the memory 1501 communicate with each other via the communication bus.
[0272] It can be seen that in the scheme provided in the embodiment of the present application, it is applied to a distributed storage system, the distributed storage system includes at least one data aggregation file, the data aggregation file is used to aggregate and store multiple data objects, and the distributed storage system can obtain the hole compression cost corresponding to the data aggregation file to be processed, wherein the data aggregation file to be processed includes valid data objects and holes, and the hole compression cost is used to represent the cost required to migrate the valid data objects to the new storage space to release the holes, write the valid data objects in the data aggregation file to be processed whose corresponding hole compression cost is not greater than the preset threshold into the new data aggregation file, and delete the data aggregation file to be processed whose corresponding hole compression cost is not greater than the preset threshold. Since the hole compression cost can represent the cost required to migrate the valid data objects to the new storage space to release the holes, then when the hole compression cost corresponding to the data aggregation file to be processed is obtained, the data aggregation files with smaller hole compression costs can be aggregated, that is, the data aggregation files with hole compression costs not greater than the preset threshold are aggregated, and the original data aggregation files are deleted after aggregation, so that the data aggregation files with smaller hole compression costs are preferentially selected for hole compression, which can achieve the purpose of consuming the least resources and releasing as much storage space as possible as quickly as possible, thereby improving the storage space utilization rate.
[0273] The communication bus mentioned in the above distributed storage system can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0274] The communication interface is used for communication between the above-mentioned distributed storage system and other devices.
[0275] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0276] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0277] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned data aggregation methods are implemented.
[0278] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any of the data aggregation methods in the above embodiments.
[0279] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.
[0280] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0281] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, distributed storage system, computer-readable storage medium, and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.
[0282] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.
Claims
1. A data aggregation method, It is characterized in that Applied to a distributed storage system, the distributed storage system includes at least one data aggregation file, the data aggregation file is used to aggregate and store multiple data objects, the method includes: Obtaining a hole compression cost corresponding to a to-be-processed data aggregate file, wherein the to-be-processed data aggregate file includes valid data objects and holes, and the hole compression cost is used to represent the cost required to migrate the valid data objects to a new storage space to release the holes; Write the valid data objects in the to-be-processed data aggregation file whose corresponding hole compression cost is not greater than a preset threshold into a new data aggregation file; Deleting the to-be-processed data aggregation file whose corresponding hole compression cost is not greater than a preset threshold; The calculation method of the hole compression cost includes: For each data aggregation file, a hole compression cost is calculated according to a relationship between a data object parameter in the data aggregation file and a cost required to migrate the valid data object to a new storage space to release the hole, wherein the data object parameter at least includes a discrete degree of the hole; The calculation method of the discrete degree of the void includes: For each data aggregation file, the holes in the data aggregation file are continuously arranged in the order of corresponding storage positions to obtain arranged continuous holes; The span between the first preset position and the second preset position in the continuous hole corresponding to the position in the data aggregation file is calculated as the discrete degree of the hole corresponding to the data aggregation file.
2. The method according to claim 1, It is characterized in that The data object parameters also include at least one of the number of valid data objects, the valid data object rate, and the size of the storage space that can be released by migrating the valid data objects; the valid data object rate is the ratio of the number of valid data objects to the total number of objects included in the data aggregation file, and the total number of objects includes the number of valid data objects and the number of holes.
3. The method according to claim 1, It is characterized in that The step of calculating the span between the positions corresponding to the first preset position and the second preset position in the continuous hole in the data aggregation file as the discrete degree of the hole corresponding to the data aggregation file includes: Normalizing the continuous holes based on the span from the first hole to the last hole in the data aggregation file; The span between the first preset position and the second preset position in the normalized continuous hole is determined as the discrete degree of the hole corresponding to the data aggregation file.
4. The method according to claim 2, It is characterized in that The step of calculating the hole compression cost according to the relationship between the data object parameters in the data aggregation file and the cost required to migrate the valid data object to the new storage space to release the hole comprises: Based on the positive correlation or negative correlation between the data object parameter and the cost required to migrate the valid data object to the new storage space to release the hole, the hole compression cost corresponding to the data aggregation file is calculated; Among them, the discreteness of the holes, the number of valid data objects and the valid data object rate are positively correlated with the cost of migrating the valid data objects to new storage space to release the holes, and the size of the storage space that can be released by migrating the valid data objects is negatively correlated with the cost of migrating the valid data objects to new storage space to release the holes.
5. The method according to claim 4, It is characterized in that The step of calculating the hole compression cost corresponding to the data aggregation file based on the positive correlation or negative correlation between the data object parameter and the cost required to migrate the valid data object to the new storage space to release the hole includes: In a case where the data object parameters include the discrete degree of the hole, the number of the valid data objects, the valid data object rate, and the size of the storage space that can be released by migrating the valid data objects, calculating a ratio between a product of a proportional parameter included in the data object parameters and an inverse parameter included in the data object parameters as a hole compression cost corresponding to the data aggregation file; The proportional parameters include the discrete degree of the holes, the number of the valid data objects and the valid data object rate, and the inverse parameters include the size of the storage space that can be released by migrating the valid data objects.
6. The method according to any one of claims 1 to 5, It is characterized in that The data aggregation file has a corresponding index aggregation file, and the index aggregation file is used to store metadata of the data object included in the corresponding data aggregation file; The method further comprises: Obtaining a data object deletion instruction, and determining an index aggregation file corresponding to the data object to be deleted indicated by the data object deletion instruction; Setting the deletion mark corresponding to the metadata of the data object to be deleted in the determined index aggregation file to a preset value, wherein the metadata corresponding to the deletion mark is the preset value and the data object corresponding to the metadata in the data aggregation file is invalid data; Count the total number of objects in the data aggregation file corresponding to the determined index aggregation file; Determine the number of valid data objects in the data aggregation file corresponding to the index aggregation file according to the number of deletion marks with a preset value in the determined index aggregation file; Based on the total number and the number of valid data objects in the data aggregation file corresponding to the index aggregation file, calculate the hole compression cost corresponding to the data aggregation file corresponding to the index aggregation file; The total number, the number of valid data objects in the data aggregation file corresponding to the index aggregation file, and the hole compression cost corresponding to the data aggregation file corresponding to the index aggregation file are recorded in the extended attributes of the index aggregation file.
7. A data aggregation device, It is characterized in that Applied to a distributed storage system, the distributed storage system includes at least one data aggregation file, the data aggregation file is used to aggregate and store multiple data objects, the device includes: A hole compression cost acquisition module, used to acquire a hole compression cost corresponding to a data aggregate file to be processed, wherein the data aggregate file to be processed includes valid data objects and holes, and the hole compression cost is used to represent the cost required to migrate the valid data objects to a new storage space to release the holes; A valid data object writing module is used to write the valid data objects in the to-be-processed data aggregation file whose corresponding hole compression cost is not greater than a preset threshold into a new data aggregation file; A file deletion module, used to delete the to-be-processed data aggregation file whose corresponding hole compression cost is not greater than a preset threshold; The device further includes a hole compression cost calculation module, which is used to calculate the hole compression cost, including: A hole compression cost calculation submodule is used to calculate the hole compression cost for each data aggregation file according to the relationship between the data object parameters in the data aggregation file and the cost required to migrate the valid data objects to the new storage space to release the holes, wherein the data object parameters at least include the discrete degree of the holes; The device further includes a discrete degree calculation module, which is used to calculate the discrete degree of the void, including: A continuous hole acquisition submodule is used to continuously arrange the holes in each data aggregation file in the order of corresponding storage positions to obtain arranged continuous holes; The discrete degree calculation submodule is used to calculate the span between the first preset position and the second preset position in the continuous hole corresponding to the position in the data aggregation file as the discrete degree of the hole corresponding to the data aggregation file.
8. A distributed storage system, It is characterized in that include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-6 when executing a program stored in a memory.
9. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
Citation Information
Patent Citations
Aggregation object recombination method and device
CN114443578A
Solid state disk buffer management method based on page replacement cost
CN115048056A
Method, apparatus and computer program product for deduplication of data
CN118244968A
