A data storage method and system

Abstract

The application provides a data storage method and system. The method comprises the following steps: determining an object category to which a to-be-stored object belongs according to the size of the to-be-stored object; if the to-be-stored object is a first type of object, sequentially storing object data and object metadata of the to-be-stored object into an index pool; if the to-be-stored object is a second type of object, determining an object level to which the to-be-stored object belongs, and storing the to-be-stored object according to the object level to which the to-be-stored object belongs; and if the to-be-stored object is a third type of object, storing object data of the to-be-stored object into a data pool, and storing object metadata of the to-be-stored object into the index pool. It can be seen that the technical scheme of the embodiment of the application can improve the space utilization rate of a disk and the storage performance of the disk.

Description

Technical Field

[0001] This application relates to the field of data storage technology, and in particular to a data storage method and system. Background Technology

[0002] With the advent of the big data era, data volume is increasing and its importance is growing. Data storage is typically done in the form of objects, known as object storage. When writing an object to object storage, the object's metadata and object data are stored separately. The object data is written directly to disk, while the object metadata is stored in a dedicated database for easy retrieval. When searching for an object, the database is first used to retrieve the corresponding object metadata, and then the object data is read from disk using the location information stored in the object metadata.

[0003] In scenarios involving the storage of massive amounts of small files, the database storing object metadata can easily become a performance bottleneck because reading and writing objects requires first manipulating their metadata. Therefore, current technologies store each file as a separate object on disk. However, since file systems have a minimum storage unit, if each object's storage size is not a multiple of this minimum unit, some disk space is wasted. Thus, a significant amount of disk space is wasted during the storage of massive amounts of small files. Furthermore, when storing files, the object metadata of all small objects is written to the index pool, causing excessive pressure on the index pool. This is especially true when the number of disks in the index pool is small compared to the number of disks in the data pool, further increasing the pressure on the index pool and degrading disk storage performance. Summary of the Invention

[0004] This application provides a data storage method and system that can improve disk space utilization and disk storage performance.

[0005] The technical solution provided in this application includes:

[0006] In a first aspect, embodiments of this application provide a data storage method, the method comprising:

[0007] The object category to which the object to be stored belongs is determined based on the size of the object to be stored.

[0008] If the object to be stored is a first type of object, then the object data and object metadata of the object to be stored are stored in the index pool in sequence; wherein, the first type of object is an object that is less than a preset first threshold.

[0009] If the object to be stored is a second type of object, then the object level to which the object to be stored belongs is determined, and the object is stored according to the object level to which the object to be stored belongs; wherein, the second type of object is an object whose file size is greater than a second threshold but less than or equal to a third threshold; the second threshold is greater than the first threshold;

[0010] If the object to be stored is a third type of object, then the object data of the object to be stored is stored in the data pool, and the object metadata of the object to be stored is stored in the index pool. The third type of object is an object whose file size is greater than a third threshold.

[0011] Secondly, embodiments of this application provide a data storage system, the system comprising:

[0012] The object category determination unit is used to determine the object category to which the object to be stored belongs based on the size of the object to be stored; if the object to be stored is a first type of object, the first storage unit is triggered; if the object to be stored is a second type of object, the second storage unit is triggered; if the object to be stored is a third type of object, the third storage unit is triggered.

[0013] The first storage unit is used to sequentially store the object data and object metadata of the object to be stored into the index pool; wherein, the first type of object is an object that is less than a preset first threshold.

[0014] The second storage unit is used to determine the object level to which the object to be stored belongs, and to store the object according to the object level to which the object to be stored belongs; wherein, the second type of object is an object whose file size is greater than a second threshold but less than or equal to a third threshold; the second threshold is greater than the first threshold;

[0015] The third storage unit is used to store the object data of the object to be stored in the data pool and the object metadata of the object to be stored in the index pool. The third type of object is an object whose file size is greater than a third threshold.

[0016] As can be seen from the above technical solutions, in this application, the object category to which the object to be stored belongs is first determined based on the size of the object to be stored. If the object to be stored is a first-class object, the object data and object metadata of the object to be stored are stored sequentially in the index pool. If the object to be stored is a second-class object, the object level to which the object to be stored belongs is determined, and the object is stored according to the object level. If the object to be stored is a third-class object, the object data of the object to be stored is stored in the data pool, and the object metadata of the object to be stored is stored in the index pool. It is evident that this application embodiment no longer processes all files in a massive file system uniformly as in the prior art, but instead determines the object category to which the object belongs based on the file size of the object, and adopts different storage measures for different object categories. This application embodiment can dynamically determine whether small objects are written to the index pool or the data pool based on the write pressure of the index pool and the data pool. When the write pressure is high, some small objects are written to the data pool, improving the write speed of small objects. In summary, this application embodiment can improve disk space utilization while also improving disk storage performance. Attached Figure Description

[0017] Figure 1 A flowchart of a data storage method provided in this application;

[0018] Figure 2 A flowchart of a data merging method provided in this application;

[0019] Figure 3 A flowchart of a data reading method provided in this application;

[0020] Figure 4 A schematic diagram of the structure of a data system provided in this application;

[0021] Figure 5 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0022] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0023] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0024] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0025] See Figure 1 , Figure 1 A flowchart of a data storage method provided in this application is shown. This method can be applied to a storage device, and the process of implementing this method may include the following steps:

[0026] Step 101: Determine the object category of the object to be stored based on its size. If the object to be stored is a first-category object, proceed to step 102; if the object to be stored is a second-category object, proceed to step 103; if the object to be stored is a third-category object, proceed to step 104.

[0027] In this embodiment, the first type of object is named merely for ease of distinction from objects described later, and is not intended to define any specific object. Similarly, the second type of object is named merely for ease of description, and is not intended to define any specific object. The third threshold is also named merely for ease of description, and is not intended to define any specific object.

[0028] In this embodiment, objects are pre-classified into at least three object categories based on their size. For each object, if its size is less than a first threshold, it is classified as a first-category object. If its size is less than a second threshold but less than or equal to a third threshold, it is classified as a second-category object. If its size is greater than the third threshold, it is classified as a third-category object. The first threshold is less than the second threshold, and the second threshold is less than the third threshold. As an example, the first threshold can be set to 256KB.

[0029] In this embodiment, the first threshold is named merely for ease of distinction from the thresholds mentioned later, and is not intended to define a specific threshold. Similarly, the second threshold is named merely for ease of description, and is not intended to define a specific threshold. The third threshold is also named merely for ease of description, and is not intended to define a specific threshold.

[0030] Step 102: Store the object data and object metadata of the object to be stored into the index pool in sequence.

[0031] As an example, the data pool uses HDD (Hard Disk Drive), while the index pool typically uses SSD (Solid State Disk or Solid State Drive).

[0032] While SSDs offer better performance than HDDs, they are also relatively more expensive. Therefore, storing small objects on an SSD can improve write performance for small objects, and because small objects don't occupy much disk space, they offer better value for money.

[0033] The object to be stored includes object data and object metadata. The object metadata stores the location information of the object data. When reading an object, the object metadata is read first, and the actual required object data is read based on the location information of the object data in the object metadata. Thus, when storing an object, the object data of the object is stored first, the location information of the object data is obtained, and the location information of the object data is stored in the object metadata, and then the object metadata is stored.

[0034] Writing all small objects to the index pool would put excessive pressure on it, especially when the index pool has fewer disks, the data pool has more disks, and there is high concurrency writing of small objects. In fact, writing small objects to an index pool composed of SSDs is not as efficient as writing to a data pool composed of HHDs. Therefore, this step stores the object metadata and object data of objects whose size belongs to the first category in the index pool.

[0035] Since the size of each object in the first type of object is lower than the first threshold, the object data and object metadata of the object to be stored are stored in the index pool. When storing the object to be stored, the object data is first stored in the index pool, and the location information of the object data of the object to be stored in the index pool is obtained and put into the object metadata. The object metadata is then stored in the index pool.

[0036] In this step, when reading and writing objects belonging to the first type of object, compared to the prior art where reading small files requires first reading the object's index data and then reading the actual object data based on the location information stored in the index data, which requires two I / O operations, the storage method provided in this application embodiment reduces one I / O operation for reading and writing object data, resulting in a significant improvement in read and write performance.

[0037] Step 103: Determine the object level to which the object to be stored belongs, and store the object according to the object level to which the object to be stored belongs.

[0038] This step involves classifying the second type of objects into different categories and implementing different storage measures for each category.

[0039] As one embodiment, one way to implement step 103 includes:

[0040] Determine whether the object to be stored belongs to an object within a specified time period and whether the number of concurrently written objects within that specified time period is less than a preset traffic threshold. If the object to be stored belongs to an object within a specified segment and the number of concurrently written objects within that specified time period is less than the preset traffic threshold, then the object to be stored is determined to be of the first level. If the object to be stored belongs to an object within a specified segment and the number of concurrently written objects within that specified time period is greater than or equal to the preset traffic threshold, then the object to be stored is determined to be of the second level.

[0041] In this embodiment, the first level is a name given for ease of distinction from the levels described later, and is not intended to define a specific level. Similarly, the second level is a name given for ease of description, and is not intended to define a specific level.

[0042] The specified time period can be understood as the time period during which concurrent writes to a small, massive file occur. Therefore, if an object to be stored belongs to this specified time period, then that object is also considered a file within this small, massive file. If the number of concurrent objects to be stored during the massive writes within the specified time period is less than a preset traffic threshold, then the object to be stored is classified as Level 1. If the number of concurrent objects to be stored during the massive writes within the specified time period is greater than or equal to the preset traffic threshold, then the object to be stored is classified as Level 2. The aforementioned traffic threshold can be set to 20.

[0043] Step 104: Store the object data of the object to be stored in the data pool, and store the object metadata of the object to be stored in the index pool.

[0044] While SSDs offer better performance than HDDs, they are also more expensive. In this step, for larger third-category objects, the object data is stored in a data pool, and the object metadata is stored in an index pool. This improves the speed of reading object metadata and reduces wasted disk space in the data pool due to small objects.

[0045] This is now complete. Figure 1 The process is shown below.

[0046] As can be seen from the above technical solutions, the object category of the object to be stored is first determined based on its size. If the object to be stored is a first-class object, its object data and object metadata are stored sequentially in the index pool. If the object to be stored is a second-class object, its object level is determined, and the object is stored according to its object level. If the object to be stored is a third-class object, its object data is stored in the data pool, and its object metadata is stored in the index pool. Therefore, this embodiment does not uniformly process all files in a massive dataset as in existing technologies. Instead, it determines the object category based on the file size of the object and adopts different storage measures for different object categories. This embodiment can dynamically determine whether small objects are written to the index pool or the data pool based on the write pressure of the index pool and the data pool. When the write pressure is high, some small objects are written to the data pool, improving the write speed of small objects. In summary, this embodiment can improve disk space utilization while also improving disk storage performance.

[0047] After completion Figure 1 Following the illustrated process, as an example, implementing step 103, which involves storing the object according to its object level, includes:

[0048] If the object level of the object to be stored is determined to be the first level, then the object data and object metadata of the object to be stored are written into the index pool in sequence.

[0049] If the object level of the object to be stored is determined to be the second level, then the current index pool weight and the data pool weight are obtained. Based on the index pool weight and the data pool weight, the storage probability of the object to be stored in the index pool and the data pool is determined, and the object data and object metadata of the object to be stored are stored according to the storage probability.

[0050] As an example, firstly, the types of each disk in the index pool and data pool are set, and the weight of each disk is calculated. For example, the weight of HDD is set to 1 and the weight of SSD is set to 10. Then, the weights of each disk in the index pool are added together to obtain the index pool weight; the weights of each disk in the data pool are added together to obtain the data pool weight.

[0051] When storing objects in the second level of storage, the data pool weight and index pool weight with pre-calculated numbers are obtained from the storage device to determine the storage probability of the object in the index pool and data pool.

[0052] For example, if the index pool weight is 1 and the data pool weight is 2, then the probability of an object being written to the index pool is (weight 1) / (weight 1 + weight 2), and the probability of it being written to the data pool is (weight 2) / (weight 1 + weight 2).

[0053] As an example, if the storage probability of the index pool is higher than that of the data pool, the object data is first stored in the index pool, and then the object metadata is stored in the index pool. If the storage probability of the index pool is lower than or equal to that of the data pool, the object data is first stored in the data pool, and then the object metadata is stored in the index pool.

[0054] As can be seen, by applying the technical solution provided in this embodiment, the object category to which an object belongs can be determined based on the file size of the object, and different storage measures can be adopted for different object categories, further improving disk space utilization and disk performance. Furthermore, the storage method provided in this embodiment can dynamically determine whether small objects are written to the index pool or the data pool based on the write pressure of the index pool and the data pool. When the write pressure is high, some small objects are written to the data pool, improving the write speed of small objects.

[0055] As an example, such as Figure 2 As shown, after storing the data to be stored, the method further includes the following steps 105 to 109:

[0056] Step 105: When the number of stored objects belonging to the second type of object and the first level is greater than the threshold for the number of small files to be merged, then proceed to step 106.

[0057] Step 106: Collect the metadata of objects belonging to the second type of objects and belonging to the first level from the index pool.

[0058] Step 107: Read the object data corresponding to each object metadata in sequence according to the collected object metadata, and merge the read object data into one object data to be rewritten into the data pool as the merged object.

[0059] In this step, for each collected object metadata, the object data is read according to the location information of the object data corresponding to the object metadata stored in the object metadata.

[0060] The object data corresponding to the read object metadata is merged into a single object data according to the merging order, and the merged object data is rewritten into the data pool as a whole, which reduces space usage. This merging order can be the order set for the object metadata before merging, or it can be the default merging order.

[0061] Step 108: Based on the location information of the merged objects in the data pool and the merging order of the objects to which the metadata of each object belonged before merging, calculate the merging location information of the objects to which the metadata of each object belongs in the data pool storing the merged objects after merging.

[0062] Step 109: Update the location information of the object data contained in the metadata of each object to the merged location information where the object data corresponding to the metadata of each object is stored, and delete the object data corresponding to the metadata of each object before merging.

[0063] This step replaces the object data location information contained in the metadata of each object before merging with the latest merged location information, so that the object data can be obtained based on the merged location information when reading object data later.

[0064] After merging is complete, the previously stored object data can be deleted to free up disk space.

[0065] This concludes the process. Figure 2 The flowchart of the system is shown.

[0066] As can be seen, the technical solution provided in this embodiment can merge stored objects that belong to the second type of objects, belong to the first level, and reach the quantity threshold, and delete the data of each object before merging after successful merging, thereby releasing disk space.

[0067] As an example, such as Figure 3 As shown, when reading a stored target object, the method further includes the following steps 110 to 112:

[0068] Step 110: Determine the object category to which the target object belongs based on its size. If the target object belongs to the first object category, proceed to step 111; otherwise, proceed to step 112.

[0069] In this step, the target object is the object to be read. If the target object does not belong to the first type of object in this step, it means that the target object belongs to other types of objects, such as the second or third type of object.

[0070] Step 111: Obtain object data from the index pool based on the object metadata of the target object stored in the index pool.

[0071] When storing the target object as described above, since the target object belongs to the first type of object, the object data and object metadata of the target object are stored in the index pool in sequence.

[0072] Step 112: Obtain the object metadata of the target object stored in the index pool, and obtain the storage location information of the object data stored on the disk based on the object metadata of the target object, so as to read the object data of the target object based on the storage location information.

[0073] If the target object belongs to the third type of object, the object data of the target object is read from the data pool based on the storage location information.

[0074] If the target object belongs to the second type of object and is at the first level, then the object data of the target object is read from the index pool based on the storage location information.

[0075] If the target object belongs to the second type of object and the second level, the object data of the target object is read from the index pool or data pool according to the storage location information.

[0076] This concludes the process. Figure 3 The flowchart of the system is shown.

[0077] It is evident that with the improved storage performance of both the data pool and the index pool, the target object can be read quickly when it is being read. Moreover, compared to existing technologies, when reading and writing first-class objects or second-class objects belonging to the first level, one IO operation for reading and writing object data is reduced, resulting in a significant improvement in read and write performance.

[0078] As an example, after reading the stored target object, the method further includes the following steps:

[0079] If the target object belongs to the first type of object, then the object metadata and object data of the target object are deleted from the index pool.

[0080] If the target object belongs to the second type of object or the third type of object before merging, then the object metadata is deleted from the index pool, and the object data of the target object is deleted from the disk.

[0081] If the target object belongs to the merged second type of object, then the object metadata is deleted from the index pool, and if it is determined that all second type objects contained in the merged object to which the target object belongs have been deleted, then the merged object is deleted.

[0082] As can be seen, this embodiment can quickly and effectively delete the target object after reading it, so as to release disk storage space in a timely manner.

[0083] See Figure 4 , Figure 4 A schematic diagram of a data storage system 400 provided in this application embodiment is shown. The system includes:

[0084] The object category determination unit 401 is used to determine the object category to which the object to be stored belongs based on the size of the object to be stored; if the object to be stored is a first type of object, the first storage unit 402 is triggered; if the object to be stored is a second type of object, the second storage unit 403 is triggered; if the object to be stored is a third type of object, the third storage unit 404 is triggered.

[0085] The first storage unit 402 is used to store the object data and object metadata of the object to be stored into the index pool in sequence; wherein, the first type of object is less than a preset first threshold.

[0086] The second storage unit 403 is used to determine the object level to which the object to be stored belongs, and to store the object according to the object level to which the object to be stored belongs; wherein, the second type of object is an object whose file size is greater than a second threshold but less than or equal to a third threshold; the second threshold is greater than the first threshold.

[0087] The third storage unit 404 is used to store the object data of the object to be stored into the data pool and the object metadata of the object to be stored into the index pool. The third type of object is an object whose file size is greater than a third threshold.

[0088] As one embodiment, the object category determination unit is specifically used for:

[0089] If the object to be stored belongs to an object within a specified time period and the number of objects to be stored concurrently written within the specified time period is less than a preset traffic threshold, then the object to be stored is determined to be of the first level.

[0090] If the object to be stored belongs to an object within a specified segment and the number of objects to be stored concurrently written within the specified time period is greater than or equal to a preset traffic threshold, then the object to be stored is determined to be of the second level.

[0091] As one embodiment, the second storage unit is specifically used for:

[0092] If the object level of the object to be stored is determined to be the first level, then the object data and object metadata of the object to be stored are written into the index pool in sequence.

[0093] If the object level of the object to be stored is determined to be the second level, then the current index pool weight and the data pool weight are obtained. Based on the index pool weight and the data pool weight, the storage probability of the object to be stored in the index pool and the data pool is determined, and the object data and object metadata of the object to be stored are stored according to the storage probability.

[0094] As one embodiment, the device further includes:

[0095] The determination unit is used to trigger the collection unit when the number of stored objects belonging to the second type of object and the first level is greater than the threshold for the number of small files to be merged.

[0096] The collection unit is used to collect metadata of objects belonging to the second type of objects and belonging to the first level that are to be merged from the index pool;

[0097] The rewrite unit is used to read the object data corresponding to each object metadata in sequence according to the collected object metadata, and merge the read object data into one object data as the merged object and rewrite it into the data pool.

[0098] The location information calculation unit is used to calculate the merge location information of the objects whose metadata belong to each object after merging in the data pool, based on the location information of the merged objects in the data pool and the merging order of the objects whose metadata belonged to each object before merging.

[0099] The update unit is used to update the location information of the object data contained in the metadata of each object to the merged location information where the object data corresponding to the metadata of each object is stored, and to delete the object data corresponding to the metadata of each object before merging.

[0100] As one embodiment, the system also includes:

[0101] The category determination unit is used to determine the object category to which the target object belongs based on the size of the target object; if the target object belongs to the first type of object, the first reading unit is triggered; if the target object does not belong to the first type of object, the second reading unit is triggered.

[0102] The first reading unit is used to retrieve the object metadata of the target object stored in the index pool and obtain object data from the index pool based on the object metadata of the target object.

[0103] The second reading unit is used to obtain the object data storage location information of the target object stored in the index pool from the object metadata of the target object, and to read the object data of the target object based on the storage location information.

[0104] As one embodiment, the system also includes:

[0105] The clearing unit is used to trigger a first deletion unit if the target object belongs to the first type of object, a second deletion unit if the target object belongs to the second type of object before merging or the third type of object, and a third deletion unit if the target object belongs to the second type of object after merging.

[0106] The first deletion unit is used to delete the object metadata and object data of the target object from the index pool.

[0107] The second deletion unit deletes object metadata from the index pool and deletes object data of the target object from the disk.

[0108] The third deletion unit deletes the object metadata from the index pool, and deletes the merged object if it is determined that all second-type objects contained in the merged object to which the target object belongs have been deleted.

[0109] As can be seen from the above technical solutions,

[0110] In this application, the object category of the object to be stored is first determined based on its size. If the object belongs to the first category, its object data and metadata are stored sequentially in the index pool. If the object belongs to the second category, its object level is determined, and the object is stored according to that level. If the object belongs to the third category, its object data is stored in the data pool, and its object metadata is stored in the index pool. Therefore, this application does not uniformly process all files in a massive dataset as in existing technologies. Instead, it determines the object category based on the file size and employs different storage measures for different object categories. This application can dynamically determine whether small objects are written to the index pool or the data pool based on the write pressure of the index pool and the data pool. When the write pressure is high, some small objects are written to the data pool, improving the write speed of small objects. In summary, this application can improve both disk space utilization and disk storage performance.

[0111] The specific implementation process of the functions and roles of each device in the above-mentioned apparatus can be found in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0112] The electronic device provided in this application, from a hardware perspective, can be found in the hardware architecture diagram. Figure 5 As shown, it includes a machine-readable storage medium and a processor, wherein: the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the data storage operations disclosed in the above example.

[0113] The machine-readable storage medium provided in this application embodiment stores machine-executable instructions. When the machine-executable instructions are invoked and executed by a processor, the machine-executable instructions cause the processor to implement the data storage operations disclosed in the above examples.

[0114] Here, a machine-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, a machine-readable storage medium can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0115] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0116] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0117] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0118] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0119] Furthermore, these computer program instructions can also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0120] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0121] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0122] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A data storage method, characterized by, The method includes: The object category to which the object to be stored belongs is determined based on the size of the object to be stored. If the object to be stored is a first type of object, then the object data and object metadata of the object to be stored are stored in the index pool in sequence; wherein, the first type of object is an object that is less than a preset first threshold. If the object to be stored is a second type of object, then if the object to be stored belongs to an object within a specified time period and the number of objects to be stored concurrently written within the specified time period is less than a preset traffic threshold, then the object to be stored is determined to be of the first level; if the object to be stored belongs to an object within a specified segment and the number of objects to be stored concurrently written within the specified time period is greater than or equal to a preset traffic threshold, then the object to be stored is determined to be of the second level. If the object level of the object to be stored is determined to be the first level, then the object data and object metadata of the object to be stored are written into the index pool sequentially; if the object level of the object to be stored is determined to be the second level, then the current index pool weight and the data pool weight are obtained, and based on the index pool weight and the data pool weight, the storage probability of the object to be stored in the index pool and the data pool is determined, and the object data and object metadata of the object to be stored are stored according to the storage probability; wherein, the second type of object is an object whose file size is greater than a second threshold but less than or equal to a third threshold; the second threshold is greater than the first threshold; If the object to be stored is a third type of object, then the object data of the object to be stored is stored in the data pool, and the object metadata of the object to be stored is stored in the index pool. The third type of object is an object whose file size is greater than a third threshold.

2. The method according to claim 1, characterized in that, After storing the data to be stored, the method further includes: When the number of stored objects belonging to the second category and the first level exceeds the threshold for merging small files, then Collect metadata of objects belonging to the second category and the first level that are to be merged from the index pool; Based on the collected object metadata, the object data corresponding to each object metadata is read sequentially, and the read object data is merged into one object data to be rewritten into the data pool as the merged object. Based on the location information of the merged objects in the data pool and the merging order of the objects to which the metadata of each object belonged before merging, calculate the merging location information of the objects to which the metadata of each object belongs in the data pool storing the merged objects after merging; Update the location information of the object data contained in the metadata of each object to the merged location information where the object data corresponding to the metadata of each object is stored, and delete the object data corresponding to the metadata of each object before merging.

3. The method as described in claim 1, characterized in that, When reading a stored target object, the method further includes: The object category to which the target object belongs is determined based on the size of the target object; If the target object belongs to the first type of object, then the object data is obtained from the index pool based on the object metadata of the target object stored in the index pool. If the target object does not belong to the first type of object, then the object metadata of the target object stored in the index pool is used to obtain the storage location information of the object data stored on the disk based on the object metadata of the target object, so as to read the object data of the target object based on the storage location information.

4. The method as described in claim 3, characterized in that, After reading the stored target object, the method further includes: If the target object belongs to the first type of object, then the object metadata and object data of the target object are deleted from the index pool; If the target object belongs to the second type of object or the third type of object before merging, then the object metadata is deleted from the index pool, and the object data of the target object is deleted from the disk; If the target object belongs to the merged second type of object, then the object metadata is deleted from the index pool, and if it is determined that all second type objects contained in the merged object to which the target object belongs have been deleted, then the merged object is deleted.

5. A data storage system, characterized in that, The system includes: The object category determination unit is used to determine the object category to which the object to be stored belongs based on the size of the object to be stored; if the object to be stored is a first type of object, the first storage unit is triggered; if the object to be stored is a second type of object, the second storage unit is triggered; if the object to be stored is a third type of object, the third storage unit is triggered. The first storage unit is used to sequentially store the object data and object metadata of the object to be stored into the index pool; wherein, the first type of object is less than a preset first threshold. The second storage unit is configured to determine the object to be stored as a first-level object if the object to be stored is a second-type object, and if the object to be stored belongs to an object within a specified time period and the number of objects to be stored concurrently written within the specified time period is less than a preset traffic threshold; and if the object to be stored belongs to an object within a specified segment and the number of objects to be stored concurrently written within the specified time period is greater than or equal to a preset traffic threshold. If the object level of the object to be stored is determined to be the first level, then the object data and object metadata of the object to be stored are written into the index pool sequentially; if the object level of the object to be stored is determined to be the second level, then the current index pool weight and the data pool weight are obtained, and based on the index pool weight and the data pool weight, the storage probability of the object to be stored in the index pool and the data pool is determined, and the object data and object metadata of the object to be stored are stored according to the storage probability; wherein, the second type of object is an object whose file size is greater than a second threshold but less than or equal to a third threshold; the second threshold is greater than the first threshold; The third storage unit is used to store the object data of the object to be stored in the data pool and the object metadata of the object to be stored in the index pool. The third type of object is an object whose file size is greater than a third threshold.

6. The system according to claim 5, characterized in that, The system also includes: The determination unit is used to trigger the collection unit when the number of stored objects belonging to the second type of object and the first level is greater than the threshold for the number of small files to be merged. The collection unit is used to collect metadata of objects belonging to the second type of objects and belonging to the first level that are to be merged from the index pool; The rewrite unit is used to read the object data corresponding to each object metadata in sequence according to the collected object metadata, and merge the read object data into one object data as the merged object and rewrite it into the data pool. The location information calculation unit is used to calculate the merge location information of the objects whose metadata belong to each object after merging in the data pool, based on the location information of the merged objects in the data pool and the merging order of the objects whose metadata belonged to each object before merging. The update unit is used to update the location information of the object data contained in the metadata of each object to the merged location information where the object data corresponding to the metadata of each object is stored, and to delete the object data corresponding to the metadata of each object before merging.

Images