Object storage method and apparatus, storage medium, and electronic device

Abstract

Embodiments of the present application provide an object storage method and device, a storage medium and an electronic device, wherein the object storage method comprises: in the case of determining that the data size of a first object received through the storage bucket is less than a first value, storing the first object into the hierarchical pool; in the case of determining that the data capacity in the hierarchical pool reaches a second value, storing a second object in the hierarchical pool into the plurality of data pools according to the storage class corresponding to the second object, wherein the second object comprises the first object stored into the hierarchical pool, and the second value is greater than the first value. Therefore, the problem that in the related art, in a mixed flash environment, a small object (equivalent to the first object with a data size less than the first value) of a storage class is directly stored into a data pool, and the storage effect is poor, can be solved. The storage effect of the small object of the storage class is improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of storage, in particular, to an object storage method and device, a storage medium and an electronic device. BACKGROUND

[0002] Distributed object storage is a scalable and cost-effective storage solution specially designed for unstructured data. With the continuous development of the Internet, the advent and use of 5G, unstructured data is growing explosively, and in addition, enterprises also hope to store various types of data for a longer period of time to facilitate more in-depth analysis. In object storage, a bucket is a logical container for storing unstructured user data, and a bucket can only be bound to one placement policy and correspond to a group of storage pools. For the same type of front-end business, the data categories may be different, some of which are hot data that needs to be frequently accessed, some of which have a lower data access frequency, and some of which are used for backup. The data in the same bucket cannot be effectively distinguished according to different storage classes, so data tiering is introduced, which proposes the concept of storage class, and the storage class is corresponded to the data pool (the storage pool includes the data pool), and there can be multiple storage classes in an object policy, and users can assign different storage classes to different data (i.e., objects) to store them in different data pools.

[0003] In the specific storage process of distributed object storage, if an object is assigned a storage class, the object will be directly stored in the data pool according to the storage class, and if an object is not assigned a storage class, the object will be stored in the hierarchical pool, and the object will be stored in the data pool through the hierarchical pool.

[0004] Since HDD is cheaper than SSD, most distributed object storage on the market currently uses a mixed flash environment, and the hierarchical pool is composed of SSD media disks, and the data pool is composed of HDD media disks. There is a significant difference between SSD (solid state disk) and HDD (mechanical hard disk) in terms of read and write data speed, and the read and write speed of SSD is much faster than that of HDD. Therefore, in the case where small objects with storage classes are directly stored in the HDD data pool, the write performance of small objects will be significantly reduced.

[0005] In addition, the minimum size of the HDD media disk when applying for storage space is 16K, and the minimum size of the SSD media disk when applying for storage space is 4K, that is, when uploading a 4K small object, if the object is to be stored in the HDD data pool, the object occupies a space size of 16K, and if the object enters the SSD hierarchical pool, the object occupies a space size of 4K. Therefore, if small objects are directly stored in the data pool, it will cause waste of storage space.

[0006] In conclusion, in the related art, there is a problem that small objects of a storage class are directly stored into a data pool in a mixed flash environment, and the storage effect is poor.

[0007] The problem that small objects of a storage class are directly stored into a data pool in a mixed flash environment and the storage effect is poor in the related art has not been effectively solved. SUMMARY

[0008] Embodiments of the present application provide an object storage method and device, a storage medium and an electronic device to at least solve the problem that small objects of a storage class are directly stored into a data pool in a mixed flash environment and the storage effect is poor in the related art.

[0009] According to an embodiment of the present application, an object storage method is provided, applied to a storage device, the storage device comprising a storage bucket, the storage bucket being connected to a plurality of data pools and a hierarchical pool respectively, the hierarchical pool being connected to the plurality of data pools, comprising: in a case where it is determined that a data size of a first object received through the storage bucket is less than a first value, storing the first object into the hierarchical pool; in a case where it is determined that a data capacity in the hierarchical pool reaches a second value, storing a second object in the hierarchical pool into the plurality of data pools according to a storage class corresponding to the second object, wherein the second object comprises the first object stored into the hierarchical pool, and the second value is greater than the first value.

[0010] In one exemplary embodiment, before storing the first object into the hierarchical pool, the method further comprises: determining a data size of the first object received through the storage bucket; in a case where the data size is greater than or equal to the first value, storing the first object into the plurality of data pools.

[0011] In an example embodiment, storing the second object into the plurality of data pools according to a storage class corresponding to the second object in the hierarchical pool comprises: determining whether the second object has a first storage class, wherein the first storage class is a storage class designated in a case that the second object is sent to the storage device; storing the second object into the plurality of data pools by the first storage class in a case that the second object has the first storage class; determining whether the storage bucket enables a first rule in a case that the second object does not have the first storage class, wherein the first rule is a matching rule used to indicate matching the second object with the plurality of data pools; storing the second object into the plurality of data pools by the first rule in a case that the storage bucket enables the first rule; determining whether the storage bucket enables a second rule in a case that the storage bucket does not enable the first rule; storing the second object into the plurality of data pools by the second rule based on state information respectively corresponding to the plurality of data pools in a case that the second rule is enabled.

[0012] In an example embodiment, storing the second object into the plurality of data pools by the first rule comprises: determining a matching condition configured by the matching rule for any data pool, wherein the matching condition comprises at least one of the following: an object prefix of a third object allowed to be stored by the any data pool, an object size of the third object, an object suffix of the third object; storing the second object satisfying the matching condition into the any data pool.

[0013] In an example embodiment, storing the second object into the plurality of data pools by the second rule based on state information respectively corresponding to the plurality of data pools comprises: obtaining the state information respectively corresponding to the plurality of data pools at a current time, wherein the current time is a time when a fourth object needs to be stored from the hierarchical pool into the plurality of data pools, the fourth object is any one of the second objects, and the state information comprises: a creation time of each data pool, a remaining capacity of the each data pool, a remaining capacity proportion of the remaining capacity occupying a capacity of the each data pool; storing the fourth object into a target data pool whose state information satisfies the second rule to store the second object into the plurality of data pools, wherein the second rule comprises at least one of the following: the earliest creation time, the largest remaining capacity, and the highest remaining capacity proportion.

[0014] In an example embodiment, storing the second object into the plurality of data pools according to a storage class corresponding to the second object in the hierarchical pool comprises: in a case where it is determined that the third rule is enabled for the storage bucket, aggregating the second object into a fifth object according to the third rule, wherein a third value corresponding to a data size of the fifth object is greater than the first value; and storing the fifth object into the plurality of data pools.

[0015] In an example embodiment, after aggregating the second object into the fifth object according to the third rule, the method further comprises: determining a proportion of a sixth object that is deleted from the second object corresponding to the fifth object relative to the second object corresponding to the fifth object; in a case where the proportion satisfies a first threshold, deleting the fifth object; and in a case where the proportion is greater than or equal to a second threshold, re-aggregating seventh objects from the second object corresponding to the fifth object except for the deleted sixth object, wherein the second threshold is less than the first threshold.

[0016] According to another embodiment of the present application, an object storage device is provided, applied to a storage device, the storage device comprising a storage bucket connected to a plurality of data pools and a hierarchical pool respectively, the hierarchical pool being connected to the plurality of data pools, comprising: a first storage module configured to, in a case where it is determined that a data size of a first object received through the storage bucket is less than a first value, store the first object into the hierarchical pool; and a second storage module configured to, in a case where it is determined that a data capacity in the hierarchical pool reaches a second value, store a second object into the plurality of data pools according to a storage class corresponding to the second object, wherein the second object comprises the first object stored into the hierarchical pool, and the second value is greater than the first value.

[0017] According to still another embodiment of the present application, a computer readable storage medium is further provided, and the computer readable storage medium stores a computer program, wherein the computer program is configured to execute steps in any of the above method embodiments when running.

[0018] According to still another embodiment of the present application, an electronic device is further provided, comprising a memory and a processor, the memory stores a computer program, and the processor is configured to execute the computer program to execute steps in any of the above method embodiments.

[0019] According to still another embodiment of the present application, a computer program product is further provided, comprising a computer program, and the computer program is executed by a processor to implement steps in any of the above method embodiments.

[0020] With the present application, a kind of object storage method is provided, applied to storage device, the storage device includes storage bucket, the storage bucket is connected to multiple data pool and hierarchical pool respectively, the hierarchical pool is connected to the multiple data pool;This method is stored in the hierarchical pool in the case where the data size of first object received by the storage bucket is less than the first value;In the case where the data capacity in the hierarchical pool reaches the second value, the second object in the hierarchical pool is stored into the multiple data pool according to the storage class corresponding to the second object, wherein the second object includes: the first object stored in the hierarchical pool, the second value is greater than the first value.Therefore, it can solve the problem that in the related art, in the mixed flash environment, there is a small object (equivalent to the first object with the data size less than the first value) of storage class, which will be directly stored into the data pool, and the storage effect is poor.Improve the storage effect of the small object of the storage class. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is a hardware structure block diagram of the computer terminal of a kind of object storage method of the embodiment of the application;

[0022] Figure 2 is a flow chart of the object storage method according to the embodiment of the application;

[0023] Figure 3 is another flow chart of the object storage method according to the embodiment of the application;

[0024] Figure 4 is still another flow chart of the object storage method according to the embodiment of the application;

[0025] Figure 5 is the structure block diagram of the object storage device according to the embodiment of the application. DETAILED DESCRIPTION

[0026] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0027] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence.

[0028] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device.For example, running on a computer terminal, Figure 1 is a hardware structure block diagram of the computer terminal of a kind of object storage method of the embodiment of the application. As shown in Figure 1 , the computer terminal can include one or more ( Figure 1Only one is shown) a processor 102 (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA and other processing devices) and a memory 104 for storing data. The computer terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. For example, the computer terminal may also include Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0029] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the load resource allocation method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0030] The transmission device 106 is used to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by a computer terminal's communications provider. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0031] Figure 2 is a flowchart of an object storage method according to an embodiment of the present application, the method being applied to a storage device, the storage device comprising storage buckets, the storage buckets being connected to a plurality of data pools and a grading pool, respectively, the grading pool being connected to the plurality of data pools; Figure 2 As shown, the process includes the following steps:

[0032] Step S202: If it is determined that the data size of the first object received through the storage bucket is less than a first value, store the first object in the classification pool;

[0033] Step S204, in a case where it is determined that the data capacity in the hierarchical pool reaches a second value, storing a second object in the plurality of data pools according to a storage class corresponding to the second object in the hierarchical pool, wherein the second object comprises the first object stored in the hierarchical pool, and the second value is greater than the first value.

[0034] It should be noted that the storage class: also known as StorageClass, can also be briefly described as SC, is the placement attribute of each object in the storage bucket. When uploading an object, the object data is placed in different storage pools according to different storage classes. Bucket: in object storage, a bucket is a logical container for storing unstructured user data.

[0035] Through the above steps, an object storage method is provided, which is applied to a storage device including a storage bucket, the storage bucket is connected to a plurality of data pools and a hierarchical pool, and the hierarchical pool is connected to the plurality of data pools. The method stores a first object in the hierarchical pool in a case where it is determined that the data size of the first object received through the storage bucket is less than a first value; and stores a second object in the plurality of data pools according to a storage class corresponding to the second object in the hierarchical pool in a case where it is determined that the data capacity in the hierarchical pool reaches a second value, wherein the second object comprises the first object stored in the hierarchical pool, and the second value is greater than the first value. Therefore, the problem that in the related art, in a mixed flash environment, a small object (equivalent to the first object with a data size less than the first value) with a storage class is directly stored in a data pool, resulting in poor storage effect, can be solved. The storage effect of the small object with the storage class is improved.

[0036] In one example embodiment, before storing the first object in the hierarchical pool, the method further comprises: determining the data size of the first object received through the storage bucket; and storing the first object in the plurality of data pools in a case where the data size is greater than or equal to the first value.

[0037] It should be noted that the small object in the embodiments of the present application generally refers to an object with a data size less than 512K. That is, the first value in the embodiments of the present application is 512K.

[0038] Therefore, the embodiments of the present application store small objects with and without a storage class in the hierarchical pool, and perform subsequent operations of storing the small objects further in the data pool through the hierarchical pool, which greatly improves the write performance of the small objects.

[0039] In an example embodiment, storing the second object into the plurality of data pools according to the storage class corresponding to the second object in the hierarchical pool comprises: determining whether the second object has a first storage class, wherein the first storage class is a storage class designated in a case that the second object is sent to the storage device; in a case that the second object has the first storage class, storing the second object into the plurality of data pools by the first storage class; in a case that the second object does not have the first storage class, determining whether the storage bucket enables a first rule, wherein the first rule is a matching rule used to indicate matching the second object with the plurality of data pools; in a case that the storage bucket enables the first rule, storing the second object into the plurality of data pools by the first rule; in a case that the storage bucket does not enable the first rule, determining whether the storage bucket enables a second rule; in a case that the second rule is enabled, storing the second object into the plurality of data pools based on state information respectively corresponding to the plurality of data pools by the second rule.

[0040] Specifically, storing the second object into the plurality of data pools by the first storage class comprises: determining second storage classes respectively corresponding to the plurality of data pools; and storing the second object into a target data pool in the plurality of data pools, wherein the second storage class of the target data pool is consistent with the first storage class of the second object. The second storage classes are pre-configured for the plurality of data pools. For example, there are three data pools, the second storage class corresponding to data pool 1 is A, the second storage class corresponding to data pool 2 is B, and the second storage class corresponding to data pool 3 is C; in a case that the second object has the first storage class and the first storage class is B, the second object is stored into data pool 2.

[0041] It should be noted that in the hierarchical pool, the priority of storing the second object into the plurality of data pools according to the first storage class of the second object is the highest; the priority of storing the second object into the plurality of data pools according to the first rule is the second highest; and the priority of storing the second object into the plurality of data pools according to the second rule is the lowest. Finally, in a case that the second object does not have the first storage class and the storage bucket is not enabled or is not configured with a third rule, the second object can be written into a default data pool in the plurality of data pools, wherein the third rule comprises at least one of the first rule and the second rule.

[0042] Therefore, by setting object storage rules with different priorities, the second object can be stored into the plurality of data pools in an orderly and efficient manner.

[0043] In an example embodiment, storing the second object into the plurality of data pools through the first rule comprises: determining that the matching rule is a matching condition configured by any data pool, wherein the matching condition comprises at least one of an object prefix of a third object allowed to be stored by the any data pool, an object size of the third object, and an object suffix of the third object; and storing the second object satisfying the matching condition into the any data pool.

[0044] Specifically, the first rule is a matching rule in the distributed object storage, and the matching rule is a bucket-level function of an object scenario. Setting the matching rule can allow data written into a bucket to be stored into different data pools according to the matching rule. Currently, the matching conditions include an object prefix, an object size, and an object suffix.

[0045] For example, data pool 1 allows storage of a third object with an object prefix D, data pool 2 allows storage of a third object with an object suffix E and an object size F, and data pool 3 allows storage of a third object with an object suffix G. Then, a second object with the object prefix D is stored into data pool 1, a second object with the object suffix E and the object size F is stored into data pool 2, and a second object with the object suffix G is stored into data pool 3.

[0046] Therefore, the embodiments of the present application can make the plurality of data pools store third objects with the same matching condition respectively through the matching rule, thereby facilitating batch processing of data stored in the data pools.

[0047] In an example embodiment, storing the second object into the plurality of data pools through the second rule based on state information corresponding to the plurality of data pools respectively comprises: obtaining the state information corresponding to the plurality of data pools respectively at a current time, wherein the current time is a time when a fourth object needs to be stored from the hierarchical pool into the plurality of data pools, and the fourth object is any object in the second object, and the state information comprises: a creation time of each data pool, a remaining capacity of the each data pool, and a remaining capacity proportion of the remaining capacity in a capacity of the each data pool; and storing the fourth object into a target data pool whose state information satisfies the second rule, to store the second object into the plurality of data pools, wherein the second rule comprises at least one of the following: the earliest creation time, the largest remaining capacity, and the highest remaining capacity proportion.

[0048] The second rule is a bucket cross-pool rule in the distributed object storage, the bucket cross-pool is short for bucket cross-pool, and the bucket cross-pool rule corresponds to a bucket-level switch of the object scenario. After the function is enabled, data written into the bucket can be stored in different data pools according to the bucket cross-pool rule. Currently, there are three kinds of bucket cross-pool rules: earliest creation time priority, largest remaining capacity priority, and highest remaining capacity ratio priority.

[0049] It can be understood that at a current moment when any object in the second object needs to be stored in multiple data pools, the state information of the multiple data pools at the current moment is determined, and a target data pool that meets the second rule is taken as a data pool for storing the any object.

[0050] In an example embodiment, storing the second object into the multiple data pools according to the storage class corresponding to the second object in the hierarchical pool comprises: in a case where it is determined that the storage bucket enables the third rule, aggregating the second object into a fifth object according to the third rule, wherein a third value corresponding to a data size of the fifth object is greater than the first value; and storing the fifth object into the multiple data pools.

[0051] The third rule is a hierarchical aggregation rule in the distributed object storage. The hierarchical aggregation rule is an object technology. After the hierarchical aggregation is enabled, objects less than or equal to 512K are first stored in a hierarchical pool when written into the cluster. When the hierarchical pool reaches a certain water level, the data is aggregated and then flushed to the data pool.

[0052] It should be noted that the aggregated fifth object is also stored into the multiple data pools according to the first storage class, the first rule, the second rule, and the like. It should also be noted that, for the first rule, the object prefix of the second object, the object size of the second object, and the object suffix of the second object can be a third storage class; for the second rule, the earliest creation time, the largest remaining capacity, and the highest remaining capacity ratio can be a fourth storage class, that is, indicating what kind (for example, the earliest creation time) of target data pool the second object needs to be stored in. Further, the storage class in the embodiment of the present application comprises: the first storage class, the third storage class, and the fourth storage class.

[0053] In an example embodiment, after the second object is aggregated into the fifth object according to the third rule, the method further comprises: determining a proportion of a deleted sixth object in the second object corresponding to the fifth object relative to the second object corresponding to the fifth object; in a case where the proportion meets a first threshold value, deleting the fifth object; and in a case where the proportion is greater than or equal to a second threshold value, re-aggregating a seventh object in the second object corresponding to the fifth object except the deleted sixth object, wherein the second threshold value is less than the first threshold value.

[0054] Optionally, the first threshold value is 100%, and the second threshold value is 70%.

[0055] Thus, the embodiments of the present application achieve the recycling of storage space by deleting or re-aggregating the fifth object according to the degree of deletion of the second object in the case that the second object corresponding to the fifth object is deleted, so as to reduce the waste of storage space.

[0056] Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and a necessary general hardware platform, and of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, an optical disk), and includes a plurality of instructions for causing a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the method described in each embodiment of the present application.

[0057] In order to better understand the process of the object storage method, the implementation process of the object storage method will be described in combination with optional embodiments below, but it is not used to limit the technical solutions of the embodiments of the present application.

[0058] At present, hierarchical aggregation and storage classes are incompatible, and the priority of the storage class is higher than that of the hierarchical aggregation. The objects with a specified storage class are stored according to the storage class, and the objects without a specified storage class are stored according to the hierarchical aggregation in the case that the hierarchical aggregation is enabled.

[0059] Based on this, the optional embodiments of the present application propose a scheme of hierarchical aggregation compatible with storage classes (equivalent to the object storage method in the above embodiments). When a small object (i.e., a first object with a data size less than a first value, and optionally, a first object with a data size equal to the first value can also be considered as a small object) is stored in a cluster, it is always stored in a hierarchical pool. After the next flush water level, the data is aggregated and flushed into the corresponding data pool according to the storage class. This can not only reduce the write performance of small objects, but also reduce the waste of storage space, and enable data to be stored in the original data pool. Specifically, it includes:

[0060] 1) Small objects enter the hierarchical pool:

[0061] In a cluster environment where hierarchical aggregation is enabled, an object (i.e., a first object) is uploaded to a bucket (i.e., a storage bucket). If the object is an object of <512K, it enters the hierarchical pool according to the hierarchical aggregation rule. If the object is an object of >=512K, it enters the data pool according to the upload rule.

[0062] 2) Open the aggregation switch, and aggregate small objects of the same storage class into large objects:

[0063] If the object is uploaded with a specified storage class (i.e., the first storage class) when uploading, a mark is recorded for the storage class. If the bucket where the object is located has a matching rule enabled, the object finds its storage class (i.e., the third storage class) according to the matching rule, and the storage class also has a mark recorded. If the bucket where the object is located has a bucket cross-pool enabled, when the small object is to be flushed, the object selects its storage pool and obtains the storage class (i.e., the fourth storage class) according to the bucket cross-pool rule.

[0064] It should be noted that the matching rule and the bucket cross-pool are functions derived from the storage class to enable data in the same bucket to be stored in different data pools according to different rules. After the bucket is set with the matching rule, the data uploaded into the bucket is compared with the matching rule. The object that meets the matching rule (i.e., the second object that meets the matching condition) is stored according to the storage class specified by the matching rule. The object that does not meet the matching rule can select a default strategy or be rejected from being written. If the bucket has the bucket cross-pool function enabled, the bucket cross-pool rule and the selected storage pool can be set. The uploaded data (i.e., the fourth object) can be selected into an optimal data pool (i.e., the target data pool) in the selected storage pool according to the bucket cross-pool strategy, and the data is stored in the data pool to obtain a storage class.

[0065] When the hierarchical pool reaches the flushing water level (i.e., the second value), the aggregation thread aggregates small objects into large objects (i.e., the fifth object) according to the storage class. Small objects of the same storage class are aggregated into multiple large objects, and then the aggregated large objects are flushed into the data pool corresponding to the storage class according to the storage class when flushing.

[0066] 3) Do not open the aggregation switch, and flush the small object into the data pool:

[0067] The storage class obtaining method is not repeated. If only the hierarchical switch is enabled and the aggregation switch is not enabled, the small object is directly flushed into the corresponding data pool according to the storage class when the hierarchical pool reaches the flushing water level.

[0068] 4) Garbage collection and defragmentation:

[0069] Garbage collection: when all the small objects corresponding to the aggregated large object (i.e., the fifth object) are deleted, the aggregated object is deleted, and the space is recycled;

[0070] Defragmentation: when the proportion of deleted small objects (i.e., the seventh object) in the aggregated large object is greater than 70%, defragmentation is performed, the small objects that have not been deleted are re-aggregated, the purpose storage class of the re-aggregation is consistent with the previous storage class, and after the re-aggregation, the original aggregated large object and the corresponding bitmap information are deleted.

[0071] Further, the following is combined with the above Figure 3The optional embodiments of the present application are explained. Specifically, data (second object) less than or equal to 512K is written into the hierarchical pool;

[0072] It is determined whether the second object in the hierarchical pool is designated a storage class (first storage class), and if yes, a storage class to which the data should be written is obtained; if no, it is determined whether the storage bucket and / or the hierarchical pool are set and a matching rule is enabled, and if yes, a storage class to which the data should be written is obtained; if the matching rule is not enabled, it is determined whether the storage bucket and / or the hierarchical pool are set and a bucket cross-pool rule is enabled, and if yes, a storage class to which the data should be written is obtained; if the bucket cross-pool rule is not enabled, the data is written into a default data pool corresponding to a default storage class.

[0073] After obtaining the storage class to which the data should be written, it is determined whether hierarchical aggregation is enabled for the storage bucket and / or the hierarchical pool, and if yes, the data is written into the hierarchical pool and a hierarchical mark is left on the bucket shard; if hierarchical aggregation is not enabled, the data is written into a data pool according to the storage class.

[0074] Further, the following is explained in combination with Figure 4 The optional embodiments of the present application are explained. Specifically, it includes: scanning the hierarchical mark on the bucket shard to determine whether the designated storage class (i.e., the first storage class) is empty; if not, the storage class is obtained; if empty, it is determined whether the matching rule or the bucket cross-pool rule is enabled, and if yes, the storage class is obtained; if the matching rule or the bucket cross-pool rule is not enabled, the storage class is the standard storage class (i.e., the default storage class);

[0075] If the hierarchical pool does not reach the lower brushing water level, the hierarchical mark on the bucket shard is continuously scanned, and if the lower brushing water level is reached, it is determined whether hierarchical aggregation is enabled, and if not, the object is brushed down to the corresponding data pool according to the obtained storage class; if hierarchical aggregation is enabled, small objects with the same storage class are aggregated into a large object; and then it is determined whether the aggregation size or the number of objects meets the requirements, and if not, the hierarchical mark on the bucket shard is continuously scanned; if yes, the object is brushed down to the corresponding data pool according to the obtained storage class.

[0076] Thus, by the optional embodiments of the present application, hierarchical aggregation is enabled, and when small objects are written, the small objects are preferentially written into the hierarchical pool, because the read and write speed of the hierarchical pool is much higher than that of the data pool, so the read and write performance of the small objects is greatly improved. In addition, after the small objects are aggregated, they are stored into the data pool, which reduces the waste of storage space.

[0077] It should be noted that in the embodiments of the present application, the small objects after opening hierarchical aggregation are first advanced to the hierarchical pool and then aggregated and flushed to the data pool corresponding to the storage class; the aggregation thread aggregates the small objects into large objects according to the storage class, and the flushing thread flushes the aggregated large objects to the data pool corresponding to the storage class according to the storage class. In this regard, it should be noted that the objects uploaded by specifying the storage class and the objects uploaded by matching the rules can be determined to be of the storage class at the time of uploading, and this storage class will be recorded on the bucket shard. The aggregation thread will scan the bucket shard to aggregate the small objects of the same storage class together. The objects uploaded by the bucket cross-pool rule can be determined to be of the storage class only when they are about to be flushed, because the capacity of the data pool may change during the period from object uploading to flushing. The optimal choice is confirmed according to the bucket cross-pool rule and the range of the pool, and the storage class is marked, and then the small objects are aggregated into large objects according to the storage class. In addition, the aggregation thread will tell the flushing thread the storage class of the aggregated large objects after aggregating the small objects into large objects. The flushing thread determines the data pool associated with the storage class according to the storage class, and then flushes the aggregated large objects into the correct data pool.

[0078] In the embodiments, an object storage device is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and details have been described above. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware or a combination of software and hardware is also possible and contemplated.

[0079] Figure 5 is a structural block diagram of an object storage device according to the embodiments of the present application, which is applied to a storage device including storage buckets, the storage buckets are respectively connected to a plurality of data pools and a hierarchical pool, and the hierarchical pool is connected to the plurality of data pools; as Figure 5 shown, the device comprises:

[0080] A first storage module 52 is configured to store a first object received through the storage bucket into the hierarchical pool when it is determined that the data size of the first object is less than a first value;

[0081] A second storage module 54 is configured to store a second object into the plurality of data pools according to the storage class corresponding to the second object when it is determined that the data capacity in the hierarchical pool reaches a second value, wherein the second object comprises the first object stored in the hierarchical pool, and the second value is greater than the first value.

[0082] The object storage method is applied to a storage device, the storage device includes a storage bucket, the storage bucket is connected to a plurality of data pools and a hierarchical pool respectively, and the hierarchical pool is connected to the plurality of data pools; the method stores a first object received through the storage bucket into the hierarchical pool in a case where a data size of the first object is determined to be less than a first value; and in a case where a data capacity in the hierarchical pool reaches a second value, the second object is stored into the plurality of data pools according to a storage class corresponding to the second object in the hierarchical pool, wherein the second object includes the first object stored into the hierarchical pool, and the second value is greater than the first value. Therefore, the problem that a small object (equivalent to the first object with the data size less than the first value) of a storage class is directly stored into a data pool in a mixed flash environment in the related art can be solved, and the storage effect of the small object of the storage class is improved.

[0083] In an example embodiment, before the first object is stored into the hierarchical pool, the device further includes a third storage module configured to determine a data size of the first object received through the storage bucket, and store the first object into the plurality of data pools in a case where the data size is greater than or equal to the first value.

[0084] In an example embodiment, the second storage module 54 is further configured to determine whether the second object has a first storage class, wherein the first storage class is a storage class designated in a case where the second object is sent to the storage device; store the second object into the plurality of data pools through the first storage class in a case where the second object has the first storage class; determine whether the storage bucket enables a first rule in a case where the second object does not have the first storage class, wherein the first rule is a matching rule used to indicate that the second object is matched with the plurality of data pools; store the second object into the plurality of data pools through the first rule in a case where the storage bucket enables the first rule; determine whether the storage bucket enables a second rule in a case where the storage bucket does not enable the first rule; and store the second object into the plurality of data pools based on state information respectively corresponding to the plurality of data pools through the second rule in a case where the second rule is enabled.

[0085] In an example embodiment, the second storage module 54 is further configured to determine a matching condition configured for any data pool by the matching rule, wherein the matching condition includes at least one of an object prefix of a third object allowed to be stored by the any data pool, an object size of the third object, and an object suffix of the third object; and store the second object satisfying the matching condition into the any data pool.

[0086] In an example embodiment, the second storage module 54 is further configured to: obtain the state information corresponding to the plurality of data pools at a current time, wherein the current time is a time when a fourth object needs to be stored from the hierarchical pool into the plurality of data pools, the fourth object is any one of the second objects, and the state information includes a creation time of each data pool, a remaining capacity of the each data pool, and a remaining capacity ratio of the remaining capacity to a capacity of the each data pool; and store the fourth object into a target data pool whose state information satisfies a second rule, to store the second object into the plurality of data pools, wherein the second rule includes at least one of the following: the earliest creation time, the largest remaining capacity, and the highest remaining capacity ratio.

[0087] In an example embodiment, the second storage module 54 is further configured to: in a case where it is determined that the storage bucket enables a third rule, aggregate the second objects into a fifth object according to the third rule, wherein a third value corresponding to a data size of the fifth object is greater than the first value; and store the fifth object into the plurality of data pools.

[0088] In an example embodiment, after the second objects are aggregated into the fifth object according to the third rule, the apparatus further includes a memory space recycling module configured to: determine a proportion of a sixth object that is deleted from the second objects corresponding to the fifth object, relative to the second objects corresponding to the fifth object; in a case where the proportion satisfies a first threshold, delete the fifth object; and in a case where the proportion is greater than or equal to a second threshold, re-aggregate seventh objects from the second objects corresponding to the fifth object except the deleted sixth object, wherein the second threshold is less than the first threshold.

[0089] Embodiments of the present application also provide a computer readable storage medium, which stores a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when running.

[0090] In an example embodiment, the above computer readable storage medium can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0091] The embodiment of the present application further provides an electronic device, comprising a memory and a processor, the memory stores a computer program, and the processor is configured to execute the computer program to perform the steps in any of the method embodiments.

[0092] In an example embodiment, the electronic device further comprises a transmission device connected to the processor and an input / output device connected to the processor.

[0093] The embodiment of the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps in any of the method embodiments.

[0094] The embodiment of the present application further provides another computer program product, which comprises a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in any of the method embodiments.

[0095] The embodiment of the present application further provides a computer program, which comprises computer instructions stored in a computer readable storage medium; a processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to perform the steps in any of the method embodiments.

[0096] The specific examples in the embodiment can refer to the examples described in the above embodiments and example embodiments, and the embodiment will not be described here.

[0097] Obviously, those skilled in the art should understand that the modules or steps of the present application described above can be realized by general computing devices, which can be concentrated on a single computing device or distributed on a network composed of multiple computing devices, and they can be realized by program codes executable by computing devices, so that they can be stored in storage devices and executed by computing devices, and in some cases, the steps shown or described can be executed in different order, or they can be manufactured into individual integrated circuit modules, or multiple modules or steps can be manufactured into a single integrated circuit module. Thus, the present application is not limited to any specific combination of hardware and software.

[0098] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. An object storage method, characterized in that: Applied to a storage device, the storage device includes storage buckets, the storage buckets are respectively connected to multiple data pools and classification pools, and the classification pools are connected to the multiple data pools. The method includes: If it is determined that the data size of the first object received through the storage bucket is less than a first value, storing the first object in the staging pool; In the case where it is determined that the data capacity in the tiered pool reaches a second value, determine whether the second object has a first storage class, wherein the first storage class is the storage class specified when the second object is sent to the storage device; in the case where the second object has the first storage class, store the second object in the multiple data pools through the first storage class; in the case where the second object does not have the first storage class, determine whether the bucket has a first rule enabled, wherein the first rule is a matching rule for instructing the second object to be matched with the multiple data pools; in the case where the bucket has the first rule enabled, store the second object in the multiple data pools through the first rule; in the case where the bucket does not have the first rule enabled, determine whether the bucket has a second rule enabled; in the case where the second rule is enabled, store the second object in the multiple data pools through the second rule based on the status information corresponding to the multiple data pools respectively; or, If it is determined that the storage bucket has enabled a third rule, aggregating the second object into a fifth object according to the third rule, wherein a third value corresponding to the data size of the fifth object is greater than the first value, wherein the third rule includes at least one of the following: the first rule, the second rule; storing the fifth object in the multiple data pools, The second object includes: the first object stored in the classification pool, the second value is greater than the first value, and after the second object is aggregated into the fifth object according to the third rule, the method further includes: determining the proportion of the deleted sixth object in the second object corresponding to the fifth object relative to the second object corresponding to the fifth object; if the proportion meets a first threshold, deleting the fifth object; and if the proportion is greater than or equal to a second threshold, re-aggregating the seventh object in the second object corresponding to the fifth object except the deleted sixth object, wherein the second threshold is less than the first threshold.

2. The method according to claim 1, characterized in that Before storing the first object in the classification pool, the method further includes: determining a data size of a first object received through the storage bucket; When the data size is greater than or equal to the first value, the first object is stored in the multiple data pools.

3. The method according to claim 1, characterized in that Storing the second object in the multiple data pools according to the first rule includes: Determining that the matching rule is a matching condition configured for any data pool, wherein the matching condition includes at least one of the following: an object prefix of a third object allowed to be stored in any data pool, an object size of the third object, and an object suffix of the third object; The second object meeting the matching condition is stored in any one of the data pools.

4. The method according to claim 1, wherein Storing the second object in the multiple data pools based on the state information respectively corresponding to the multiple data pools according to the second rule includes: Obtaining the status information corresponding to each of the multiple data pools at a current moment, wherein the current moment is a moment when a fourth object needs to be stored from the hierarchical pool to the multiple data pools, and the fourth object is any object among the second objects, wherein the status information includes: a creation time of each data pool, a remaining capacity of each data pool, and a remaining capacity ratio of the remaining capacity to the capacity of each data pool; The fourth object is stored in a target data pool whose status information satisfies the second rule, so as to store the second object in the multiple data pools, wherein the second rule includes at least one of the following: earliest creation time, largest remaining capacity, and highest remaining capacity ratio.

5. An object storage device, characterized in that: Applied to a storage device, the storage device includes a storage bucket, the storage bucket is respectively connected to a plurality of data pools and a classification pool, the classification pool is connected to the plurality of data pools, The device comprises: a first storage module, configured to store the first object in the classification pool if it is determined that the data size of the first object received through the storage bucket is less than a first value; The second storage module is configured to, when it is determined that the data capacity in the classification pool reaches a second value, determine whether the second object has a first storage class, wherein the first storage class is a storage class specified when the second object is sent to the storage device; when the second object has the first storage class, store the second object in the multiple data pools through the first storage class; when the second object does not have the first storage class, determine whether the storage bucket has a first rule enabled, wherein the first rule is a matching rule for instructing the second object to be matched with the multiple data pools; when the storage bucket has the first rule enabled, store the second object in the multiple data pools through the first rule. Then, the second object is stored in the multiple data pools; if the first rule is not enabled in the storage bucket, it is determined whether the second rule is enabled in the storage bucket; if the second rule is enabled, the second object is stored in the multiple data pools based on the status information corresponding to the multiple data pools respectively through the second rule; or, the second storage module is further used to, if it is determined that the third rule is enabled in the storage bucket, aggregate the second object into a fifth object according to the third rule, wherein a third value corresponding to the data size of the fifth object is greater than the first value, wherein the third rule includes at least one of the following: the first rule, the second rule; storing the fifth object in the multiple data pools, Wherein, the second object includes: the first object stored in the classification pool, the second value is greater than the first value, wherein the device also includes a memory space recovery module, used to determine the proportion of the deleted sixth object in the second object corresponding to the fifth object relative to the second object corresponding to the fifth object; if the proportion meets a first threshold, delete the fifth object; if the proportion is greater than or equal to a second threshold, re-aggregate the seventh object in the second object corresponding to the fifth object except the deleted sixth object, wherein the second threshold is less than the first threshold.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 4 when executed by a processor.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

Images