Object storage method and device, storage medium and electronic equipment

Abstract

The embodiment of the invention provides an object storage method and device, a storage medium and electronic device.The object storage method comprises the steps that under the condition that it is determined that the data size of a first object received through a storage bucket is smaller than a first value, the first object is stored in a classification pool; when it is determined that the data capacity in the hierarchical pool reaches a second value, a second object is stored in the multiple data pools according to a storage class corresponding to the second object in the hierarchical pool, the second object comprises the first object stored in the hierarchical pool, and the second value is larger than the first value. Therefore, the problem that in the related technology, in the mixed flash environment, small objects (equivalent to the first object with the data size smaller than the first value) of storage classes can be directly stored in the data pool, and the storage effect is poor can be solved. And the storage effect of the small object with the storage class is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of storage technology, and in particular, to an object storage method and device, a storage medium, and an electronic device. Background Art

[0002] Distributed object storage is a massively scalable and cost-effective storage solution for unstructured data. With the continuous development of the Internet and the arrival and use of 5G, unstructured data continues to grow explosively. In addition, enterprises also hope to store various types of data for longer and longer periods of time for more in-depth research and analysis. In object storage, a bucket is a logical container for storing unstructured user data. A bucket can only be bound to one placement strategy and corresponds to a group of storage pools. The categories of data generated by the same front-end business may be different. Some are hot data that needs to be accessed frequently, some are accessed less frequently, and some are used for backup. Data of different storage categories in the same bucket cannot be effectively distinguished at present, so data tiering came into being. It proposes the concept of storage class, which corresponds storage class to data pool (storage pool includes data pool). There can be multiple storage classes in an object strategy. Users can specify different storage classes for different data (i.e. objects) so that these data can be stored in different data pools.

[0003] In the specific storage process of distributed object storage, if an object is assigned a storage class, the object will be directly stored in the data pool according to the storage class. If an object is not assigned a storage class, the object will be stored in the classification pool and then stored in the data pool through the classification pool.

[0004] Since HDD is cheaper than SSD, most of the distributed object storage on the market currently uses a mixed flash environment. The grading pool is composed of SSD media disks, and the data pool is composed of HDD media disks. There is a significant difference in the reading and writing data speeds between SSD (solid state drive) and HDD (mechanical hard disk). The reading and writing speeds of SSD are much faster than those of HDD, so the reading and writing speeds of the grading pool are much faster than those of the data pool. Therefore, when small objects of the storage class are directly stored in the HDD data pool, the write performance of small objects will be greatly reduced.

[0005] In addition, the minimum storage space requested for HDD media is 16K, and the minimum storage space requested for SSD media is 4K. That is to say, when uploading a small 4K object, if this object is to be stored in the HDD data pool, the space it occupies is 16K. If this object enters the SSD classification pool, the space it occupies is 4K. Therefore, if the small object is directly stored in the data pool, it will lead to a waste of storage space.

[0006] In summary, in the related technology, in a mixed flash environment, small objects of the storage class will be directly stored in the data pool, resulting in poor storage effect.

[0007] Regarding the related technologies, in a mixed flash environment, small objects of the storage class will be directly stored in the data pool, and the problem of poor storage effect has not been effectively solved. Summary of the invention

[0008] The embodiments of the present application provide an object storage method and device, a storage medium, and an electronic device to at least solve the problem in the related art that in a mixed flash environment, small objects of the storage class will be directly stored in a data pool, resulting in poor storage effect.

[0009] According to an embodiment of the present application, there is provided an object storage method, which is applied to a storage device, wherein the storage device comprises a storage bucket, wherein the storage bucket is respectively connected to a plurality of data pools and a classification pool, wherein the classification pool is connected to the plurality of data pools, and comprises: upon determining that a data size of a first object received through the storage bucket is less than a first value, storing the first object in the classification pool; upon determining that a data capacity in the classification pool reaches a second value, storing the second object in the classification pool in the plurality of data pools according to a storage class corresponding to the second object, wherein the second object comprises: the first object stored in the classification pool, and the second value is greater than the first value.

[0010] In an exemplary embodiment, before storing the first object in the classification pool, the method further includes: determining a data size of the first object received through the storage bucket; and storing the first object in the multiple data pools if the data size is greater than or equal to the first value.

[0011] In an exemplary embodiment, storing the second object in the classification pool into the multiple data pools according to the storage class corresponding to the second object includes: determining whether the second object has a first storage class, wherein the first storage class is a storage class specified when the second object is sent to the storage device; if the second object has the first storage class, storing the second object in the multiple data pools through the first storage class; if the second object does not have the first storage class, determining whether the bucket has a first rule enabled, wherein the first rule is a matching rule for indicating that the second object is matched with the multiple data pools; if the bucket has the first rule enabled, storing the second object in the multiple data pools through the first rule; if the bucket does not have the first rule enabled, determining whether the bucket has the second rule enabled; if the second rule is enabled, storing the second object in the multiple data pools through the second rule based on status information corresponding to the multiple data pools.

[0012] In an exemplary embodiment, storing the second object in the multiple data pools through the first rule includes: determining the matching conditions configured by the matching rule for any data pool, wherein the matching conditions include at least one of the following: an object prefix of a third object allowed to be stored in any data pool, an object size of the third object, and an object suffix of the third object; and storing the second object that meets the matching conditions in any data pool.

[0013] In an exemplary embodiment, storing the second object in the multiple data pools based on the status information respectively corresponding to the multiple data pools through the second rule includes: obtaining the status information respectively corresponding to the multiple data pools at the current moment, wherein the current moment is the moment when the fourth object needs to be stored from the classification pool to the multiple data pools, and the fourth object is any object among the second objects, wherein the status information includes: the creation time of each data pool, the remaining capacity of each data pool, and the ratio of the remaining capacity to the remaining capacity of each data pool; storing the fourth object in the target data pool whose status information satisfies the second rule, so as to store the second object in the multiple data pools, wherein the second rule includes at least one of the following: the earliest creation time, the largest remaining capacity, and the highest remaining capacity ratio.

[0014] In an exemplary embodiment, storing the second object in the classification pool into the multiple data pools according to the storage class corresponding to the second object includes: when determining that the third rule is enabled for the storage bucket, aggregating the second object into a fifth object according to the third rule, wherein a third value corresponding to the data size of the fifth object is greater than the first value; and storing the fifth object into the multiple data pools.

[0015] In an exemplary embodiment, after aggregating the second objects into the fifth object according to the third rule, the method further includes: determining the proportion of the deleted sixth object in the second object corresponding to the fifth object relative to the second object corresponding to the fifth object; deleting the fifth object when the proportion meets a first threshold; and re-aggregating the seventh object in the second object corresponding to the fifth object except the deleted sixth object when the proportion is greater than or equal to a second threshold, wherein the second threshold is less than the first threshold.

[0016] According to another embodiment of the present application, an object storage device is provided, which is applied to a storage device, wherein the storage device includes storage buckets, wherein the storage buckets are respectively connected to multiple data pools and classification pools, wherein the classification pools are connected to the multiple data pools, and include: a first storage module, which is used to store the first object in the classification pool when it is determined that the data size of the first object received through the storage bucket is less than a first value; and a second storage module, which is used to store the second object in the classification pool according to the storage class corresponding to the second object in the classification pool when it is determined that the data capacity in the classification pool reaches a second value, wherein the second object includes: the first object stored in the classification pool, and the second value is greater than the first value.

[0017] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when run.

[0018] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0019] According to another embodiment of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0020] Through the present application, an object storage method is provided, which is applied to a storage device, wherein the storage device includes a storage bucket, wherein the storage bucket is respectively connected to a plurality of data pools and a classification pool, wherein the classification pool is connected to the plurality of data pools; when it is determined that the data size of the first object received through the storage bucket is less than a first value, the method stores the first object in the classification pool; when it is determined that the data capacity in the classification pool reaches a second value, the second object is stored in the plurality of data pools according to the storage class corresponding to the second object in the classification pool, wherein the second object includes: the first object stored in the classification pool, and the second value is greater than the first value. Therefore, the problem in the related art that in a mixed flash environment, small objects (equivalent to the first object whose data size is less than the first value) in the storage class will be directly stored in the data pool, resulting in poor storage effect, can be solved. The storage effect of small objects in the storage class is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a hardware structure block diagram of a computer terminal of an object storage method according to an embodiment of the present application;

[0022] Figure 2 is a flowchart of an object storage method according to an embodiment of the present application;

[0023] Figure 3 is another flow chart of the object storage method according to an embodiment of the present application;

[0024] Figure 4 is another flow chart of the object storage method according to an embodiment of the present application;

[0025] Figure 5 It is a structural block diagram of an object storage device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0028] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 1 is a hardware structure block diagram of a computer terminal of an object storage method according to an embodiment of the present application. Figure 1 As shown, the computer terminal may include one or more ( Figure 1Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the above-mentioned computer terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0029] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for allocating load resources in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above method is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0030] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of a computer terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NlC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0031] Figure 2 is a flowchart of an object storage method according to an embodiment of the present application, the method is applied to a storage device, the storage device includes a storage bucket, the storage bucket is respectively connected to a plurality of data pools and a classification pool, the classification pool is connected to the plurality of data pools; Figure 2 As shown, the process includes the following steps:

[0032] Step S202: if it is determined that the data size of the first object received through the storage bucket is less than a first value, store the first object in the classification pool;

[0033] Step S204, when it is determined that the data capacity in the classification pool reaches a second value, the second object in the classification pool is stored in the multiple data pools according to the storage class corresponding to the second object, wherein the second object includes: the first object stored in the classification pool, and the second value is greater than the first value.

[0034] It should be noted that the storage class, also known as StorageClass, can also be abbreviated as SC, is the placement attribute of each object in the bucket. When uploading an object, the object data is placed in different storage pools according to different storage classes. Bucket: In object storage, a bucket is a logical container for storing unstructured user data.

[0035] Through the above steps, an object storage method is provided, which is applied to a storage device, wherein the storage device includes a storage bucket, wherein the storage bucket is respectively connected to a plurality of data pools and a classification pool, wherein the classification pool is connected to the plurality of data pools; when it is determined that the data size of the first object received through the storage bucket is less than a first value, the method stores the first object in the classification pool; when it is determined that the data capacity in the classification pool reaches a second value, the second object is stored in the plurality of data pools according to the storage class corresponding to the second object in the classification pool, wherein the second object includes: the first object stored in the classification pool, and the second value is greater than the first value. Therefore, the problem in the related art that in a mixed flash environment, small objects (equivalent to the first object whose data size is less than the first value) in the storage class will be directly stored in the data pool, resulting in poor storage effect, can be solved. The storage effect of small objects in the storage class is improved.

[0036] In an exemplary embodiment, before storing the first object in the classification pool, the method further includes: determining a data size of the first object received through the storage bucket; and storing the first object in the multiple data pools if the data size is greater than or equal to the first value.

[0037] It should be noted that the small object in the embodiment of the present application generally refers to an object whose data size is less than 512K, that is, the first value in the embodiment of the present application is 512K.

[0038] Therefore, the embodiment of the present application greatly improves the writing performance of small objects by storing both small objects with storage classes and small objects without storage classes in the classification pool, and performing subsequent operations of further storing the small objects in the data pool through the classification pool.

[0039] In an exemplary embodiment, storing the second object in the classification pool into the multiple data pools according to the storage class corresponding to the second object includes: determining whether the second object has a first storage class, wherein the first storage class is a storage class specified when the second object is sent to the storage device; if the second object has the first storage class, storing the second object in the multiple data pools through the first storage class; if the second object does not have the first storage class, determining whether the bucket has a first rule enabled, wherein the first rule is a matching rule for indicating that the second object is matched with the multiple data pools; if the bucket has the first rule enabled, storing the second object in the multiple data pools through the first rule; if the bucket does not have the first rule enabled, determining whether the bucket has the second rule enabled; if the second rule is enabled, storing the second object in the multiple data pools through the second rule based on status information corresponding to the multiple data pools.

[0040] Specifically, storing the second object in the multiple data pools through the first storage class includes: determining the second storage classes respectively corresponding to the multiple data pools; storing the second object in a target data pool among the multiple data pools, wherein the second storage class of the target data pool is consistent with the first storage class of the second object. The second storage class is pre-configured for the multiple data pools. For example, there are three data pools, the second storage class corresponding to data pool 1 is A, the second storage class corresponding to data pool 2 is B, and the second storage class corresponding to data pool 3 is C; then, if the second object has the first storage class and the first storage class is B, the second object is stored in data pool 2.

[0041] It should be noted that, in the hierarchical pool, storing the second object in multiple data pools according to the first storage class of the second object has the highest priority; the second priority is storing the second object in multiple data pools according to the first rule; the lowest priority is storing the second object in multiple data pools according to the second rule. Finally, in the case where the second object does not have the first storage class and the storage bucket is not enabled or the third rule is not configured, the second object can be written to the default data pool in the multiple data pools; wherein the third rule includes at least one of the following: the first rule, the second rule.

[0042] Therefore, the embodiment of the present application can store the second object in multiple data pools in an orderly and efficient manner by setting object storage rules with different priorities.

[0043] In an exemplary embodiment, storing the second object in the multiple data pools through the first rule includes: determining the matching conditions configured by the matching rule for any data pool, wherein the matching conditions include at least one of the following: an object prefix of a third object allowed to be stored in any data pool, an object size of the third object, and an object suffix of the third object; and storing the second object that meets the matching conditions in any data pool.

[0044] Specifically, the first rule is the matching rule in distributed object storage. The matching rule is a bucket-level function for object scenarios. Setting the matching rule allows the data written into the bucket to be stored in different data pools according to the matching rule. Currently, there are three matching conditions: object prefix, object size, and object suffix.

[0045] Exemplarily, data pool 1 allows storage of a third object with an object prefix of D, data pool 2 allows storage of a third object with an object suffix of E and an object size of F, and data pool 3 allows storage of a third object with an object suffix of G; then the second object with an object prefix of D is stored in data pool 1, the second object with an object suffix of E and an object size of F is stored in data pool 2, and the second object with an object suffix of G is stored in data pool 3.

[0046] Therefore, the embodiment of the present application can enable multiple data pools to respectively store third objects with the same matching conditions through matching rules, thereby facilitating batch processing of the data stored in the data pools.

[0047] In an exemplary embodiment, storing the second object in the multiple data pools based on the status information respectively corresponding to the multiple data pools through the second rule includes: obtaining the status information respectively corresponding to the multiple data pools at the current moment, wherein the current moment is the moment when the fourth object needs to be stored from the classification pool to the multiple data pools, and the fourth object is any object among the second objects, wherein the status information includes: the creation time of each data pool, the remaining capacity of each data pool, and the ratio of the remaining capacity to the remaining capacity of each data pool; storing the fourth object in the target data pool whose status information satisfies the second rule, so as to store the second object in the multiple data pools, wherein the second rule includes at least one of the following: the earliest creation time, the largest remaining capacity, and the highest remaining capacity ratio.

[0048] The second rule is the bucket cross-pool rule in distributed object storage. Bucket cross-pool is the abbreviation of bucket cross-data pool. Bucket cross-pool rule corresponds to a bucket-level switch for object scenarios. After this function is turned on, the data written into the bucket can be stored in different data pools according to the bucket cross-pool rule. There are currently three types of bucket cross-pool rules: the earliest creation time is prioritized, the largest remaining capacity is prioritized, and the highest remaining capacity ratio is prioritized.

[0049] It can be understood that when any object in the second object needs to be stored in multiple data pools, the status information of the multiple data pools at the current moment is determined, and the target data pool whose status information meets the second rule is used as the data pool for storing the any object.

[0050] In an exemplary embodiment, storing the second object in the classification pool into the multiple data pools according to the storage class corresponding to the second object includes: when determining that the third rule is enabled for the storage bucket, aggregating the second object into a fifth object according to the third rule, wherein a third value corresponding to the data size of the fifth object is greater than the first value; and storing the fifth object into the multiple data pools.

[0051] The third rule is the hierarchical aggregation rule in distributed object storage. The hierarchical aggregation rule is an object technology. After hierarchical aggregation is enabled, objects less than or equal to 512K will be first written to the hierarchical pool when they are written to the cluster. After the hierarchical pool reaches a certain water level, the data will be aggregated and flushed to the data pool.

[0052] It should be noted that the aggregated fifth object is also stored in multiple data pools according to the first storage class, the first rule, the second rule, etc. It should also be noted that for the first rule, the object prefix of the second object, the object size of the second object, and the object suffix of the second object can be the third storage class; for the second rule, the earliest creation time, the largest remaining capacity, and the highest remaining capacity ratio can be the fourth storage class, that is, indicating what kind of target data pool (for example, the earliest creation time) the second object needs to be stored in. Furthermore, the storage classes in the embodiments of the present application include: the first storage class, the third storage class, and the fourth storage class.

[0053] In an exemplary embodiment, after aggregating the second objects into the fifth object according to the third rule, the method further includes: determining the proportion of the deleted sixth object in the second object corresponding to the fifth object relative to the second object corresponding to the fifth object; deleting the fifth object when the proportion meets a first threshold; and re-aggregating the seventh object in the second object corresponding to the fifth object except the deleted sixth object when the proportion is greater than or equal to a second threshold, wherein the second threshold is less than the first threshold.

[0054] Optionally, the first threshold is 100% and the second threshold is 70%.

[0055] Therefore, in the embodiment of the present application, when the second object corresponding to the fifth object is deleted, the fifth object is deleted or re-aggregated according to the degree of deletion of the second object, thereby realizing storage space recovery and reducing waste of storage space.

[0056] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0057] In order to better understand the process of the above-mentioned object storage method, the implementation process of the above-mentioned object storage method is described below in combination with an optional embodiment, but it is not used to limit the technical solution of the embodiment of the present application.

[0058] Currently, hierarchical aggregation and storage class are incompatible and storage class has a higher priority than hierarchical aggregation. Objects with specified storage class are stored according to storage class. Objects without specified storage class are stored according to hierarchical aggregation when hierarchical aggregation is enabled.

[0059] Based on this, the optional embodiment of the present application proposes a hierarchical aggregation compatible storage class solution (equivalent to the object storage method in the above embodiment). When a small object (i.e., a first object whose data size is less than a first value, optionally, a first object whose data size is equal to the first value can also be considered a small object) is stored in the cluster, it is all stored in the hierarchical pool. After the water level is lowered, the data is aggregated and flushed into the corresponding data pool according to the storage class. This can not only avoid reducing the write performance of small objects, but also reduce the waste of storage space, and allow the data to be stored in the data pool where it was originally intended to be stored. Specifically, it includes:

[0060] 1) Small objects enter the classification pool:

[0061] In a cluster environment with hierarchical aggregation enabled, an object (i.e., the first object) is uploaded to a bucket (i.e., a storage bucket). If the object is less than 512K, it enters the hierarchical pool according to the hierarchical aggregation rules. Objects >= 512K enter the data pool according to the upload rules.

[0062] 2) Turn on the aggregation switch to aggregate small objects of the same storage class into large objects:

[0063] If the object is uploaded with a specified storage class (i.e., the first storage class), a tag will be created to record the storage class. If the bucket where the object is located has matching rules enabled, the object will find its own storage class (i.e., the third storage class) based on the matching rules, and this storage class will also be recorded with a tag. If the bucket where the object is located has bucket cross-pool enabled, then when the small object is to be flushed, it will select its own storage pool and obtain the storage class (i.e., the fourth storage class) based on the bucket cross-pool rules.

[0064] It should be noted that matching rules and bucket cross-pool are functions derived from storage classes that allow data in the same bucket to be stored in different data pools according to different rules. After the bucket is set with matching rules, the data uploaded into the bucket is compared with the matching rules. Objects that meet the matching rules (i.e., the second object that meets the matching conditions) are stored according to the storage class specified by the matching rules. Objects that do not meet the matching rules can select the default policy or refuse to write. If the bucket cross-pool function is enabled for the bucket, you can set the bucket cross-pool rules and select the storage pool. The uploaded data (i.e., the fourth object) can select an optimal data pool (i.e., the target data pool) in the selected storage pool according to the bucket cross-pool policy, store the data in it, and obtain a storage class.

[0065] After the classification pool reaches the flushing water level (i.e., the second value), the aggregation thread will aggregate small objects into large objects (i.e., the fifth object) according to the storage class, and small objects of the same storage class will be aggregated into multiple large objects. Then, when flushing, the aggregated large objects will be flushed into the data pool corresponding to the storage class according to the storage class.

[0066] 3) Do not turn on the aggregation switch and flush small objects into the data pool:

[0067] The method of obtaining storage classes is not described here. If only the classification switch is turned on but the aggregation switch is not turned on, then when the classification pool reaches the flushing water level, small objects are directly flushed into the corresponding data pool according to the storage class.

[0068] 4) Garbage collection and defragmentation:

[0069] Garbage collection: When all small objects corresponding to the entire aggregate large object (i.e., the fifth object) are deleted, the aggregate object is deleted and space is reclaimed;

[0070] Defragmentation: When the percentage of deleted small objects corresponding to the aggregated large object is greater than 70%, defragmentation is performed and the small objects that have not been deleted (i.e., the seventh object) are re-aggregated. The destination storage class of the aggregation remains consistent with the previous one. After re-aggregation, the original aggregated large object and the corresponding bitmap information are deleted.

[0071] Furthermore, the following combination Figure 3The optional embodiment of the present application is explained. Specifically, data (second object) less than or equal to 512K is written into the classification pool;

[0072] Determine whether the second object in the classification pool is assigned a storage class (first storage class). If so, obtain the storage class to which the data should be written; if not, determine whether the bucket and / or classification pool is set and the matching rule is enabled. If the matching rule is enabled, obtain the storage class to which the data should be written; if the matching rule is not enabled, determine whether the bucket and / or classification pool is set and the bucket cross-pool rule is enabled. If the bucket cross-pool rule is enabled, obtain the storage class to which the data should be written; if the bucket cross-pool rule is enabled, write the data to the default data pool corresponding to the default storage class.

[0073] After obtaining the storage class to which the data should be written, determine whether the bucket and / or tiered pool has tiered aggregation enabled. If enabled, the data is written to the tiered pool and a tiered mark is left on the bucket shard; if tiered aggregation is not enabled, the data is written to the data pool according to the storage class.

[0074] Furthermore, the following combination Figure 4 The optional embodiments of the present application are explained. Specifically, it includes: scanning the classification mark on the bucket shard to determine whether the specified storage class (i.e., the first storage class) is empty; if it is not empty, the storage class is obtained; if it is empty, it is determined whether the matching rule or the bucket cross-pool rule is enabled, and if the matching rule or the bucket cross-pool rule is enabled, the storage class is obtained; if the matching rule or the bucket cross-pool rule is not enabled, the storage class is the standard storage class (i.e., the default storage class);

[0075] If the flushing water level is not reached in the grading pool, the grading mark on the bucket shard will continue to be scanned. If the flushing water level is reached, it is determined whether to enable grading aggregation. If not, objects are flushed to the corresponding data pool according to the acquired storage class. If grading aggregation is enabled, small objects of the same storage class are aggregated into large objects. It is then determined whether the aggregation size or number is met. If the aggregation size or number is not met, the grading mark on the bucket shard will continue to be scanned. If it is met, the grading mark on the bucket shard will continue to be scanned, and objects are flushed to the corresponding data pool according to the acquired storage class.

[0076] Therefore, through the optional embodiment of the present application, hierarchical aggregation is enabled, and when small objects are written, the small objects will be written into the hierarchical pool first, because the reading and writing speed of the hierarchical pool is much higher than that of the data pool, so the reading and writing performance of small objects will be greatly improved. In addition, aggregating small objects and then storing them in the data pool will reduce the waste of storage space.

[0077] It should be noted that, in the embodiment of the present application, after the hierarchical aggregation is turned on, the small objects are first put into the hierarchical pool and then aggregated and brushed to the data pool corresponding to the storage class; the aggregation thread aggregates the small objects into large objects according to the storage class, and the brushing thread brushes the aggregated large objects into the data pool corresponding to the storage class according to the storage class. It should be noted that the objects uploaded by the specified storage class and the objects uploaded by the matching rules can already determine the storage class when uploading, and this storage class will be recorded on the bucket shards. The aggregation thread will scan the bucket shards to aggregate the small objects with the same storage class together. The objects uploaded by the bucket cross-pool will not be determined until the storage class is about to be brushed, because the capacity of the data pool may change during the period from the object being uploaded to the brushing. According to the bucket cross-pool rules and the range of the pool, the optimal choice is confirmed and the storage class is marked, and then the small objects are aggregated into large objects according to the storage class. In addition, after the aggregation thread aggregates the small objects into large objects, it will tell the brushing thread the storage class of these aggregated large objects. The brushing thread determines the data pool associated with the storage class according to the storage class, and then brushes the aggregated large object into the correct data pool.

[0078] In this embodiment, an object storage device is also provided, which is used to implement the above embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0079] Figure 5 is a structural block diagram of an object storage device according to an embodiment of the present application, the device is applied to a storage device, the storage device includes a storage bucket, the storage bucket is respectively connected to a plurality of data pools and a classification pool, the classification pool is connected to the plurality of data pools; Figure 5 As shown, the device comprises:

[0080] A first storage module 52, configured to store the first object in the classification pool if it is determined that the data size of the first object received through the storage bucket is less than a first value;

[0081] The second storage module 54 is used to store the second object in the classification pool into the multiple data pools according to the storage class corresponding to the second object in the classification pool when it is determined that the data capacity in the classification pool has reached a second value, wherein the second object includes: the first object stored in the classification pool, and the second value is greater than the first value.

[0082] Through the above-mentioned device, an object storage method is provided, which is applied to a storage device, wherein the storage device includes a storage bucket, wherein the storage bucket is respectively connected to a plurality of data pools and a classification pool, wherein the classification pool is connected to the plurality of data pools; when it is determined that the data size of the first object received through the storage bucket is less than a first value, the method stores the first object in the classification pool; when it is determined that the data capacity in the classification pool reaches a second value, the second object is stored in the plurality of data pools according to the storage class corresponding to the second object in the classification pool, wherein the second object includes: the first object stored in the classification pool, and the second value is greater than the first value. Therefore, the problem in the related art that in a mixed flash environment, small objects (equivalent to the first object whose data size is less than the first value) in the storage class will be directly stored in the data pool, resulting in poor storage effect, can be solved. The storage effect of small objects in the storage class is improved.

[0083] In an exemplary embodiment, before storing the first object in the classification pool, the device also includes a third storage module, which is used to: determine the data size of the first object received through the storage bucket; and store the first object in the multiple data pools when the data size is greater than or equal to the first value.

[0084] In an exemplary embodiment, the second storage module 54 is also used to: determine whether the second object has a first storage class, wherein the first storage class is the storage class specified when the second object is sent to the storage device; if the second object has the first storage class, store the second object in the multiple data pools through the first storage class; if the second object does not have the first storage class, determine whether the storage bucket has a first rule enabled, wherein the first rule is a matching rule for indicating that the second object is matched with the multiple data pools; if the storage bucket has the first rule enabled, store the second object in the multiple data pools through the first rule; if the storage bucket does not have the first rule enabled, determine whether the storage bucket has a second rule enabled; if the second rule is enabled, store the second object in the multiple data pools through the second rule based on status information corresponding to the multiple data pools.

[0085] In an exemplary embodiment, the second storage module 54 is also used to: determine the matching conditions configured for any data pool by the matching rule, wherein the matching conditions include at least one of the following: an object prefix of a third object allowed to be stored in any data pool, an object size of the third object, and an object suffix of the third object; and store the second object that meets the matching conditions in any data pool.

[0086] In an exemplary embodiment, the second storage module 54 is also used to: obtain the status information corresponding to the multiple data pools at the current moment, wherein the current moment is the moment when the fourth object needs to be stored from the classification pool to the multiple data pools, and the fourth object is any object among the second objects, wherein the status information includes: the creation time of each data pool, the remaining capacity of each data pool, and the ratio of the remaining capacity to the remaining capacity of each data pool; store the fourth object in the target data pool whose status information meets the second rule, so as to store the second object in the multiple data pools, wherein the second rule includes at least one of the following: the earliest creation time, the largest remaining capacity, and the highest remaining capacity ratio.

[0087] In an exemplary embodiment, the second storage module 54 is also used to: when it is determined that the third rule is enabled for the storage bucket, aggregate the second object into a fifth object according to the third rule, wherein a third value corresponding to the data size of the fifth object is greater than the first value; and store the fifth object in the multiple data pools.

[0088] In an exemplary embodiment, after aggregating the second objects into the fifth object according to the third rule, the device also includes a memory space recovery module, which is used to: determine the proportion of the deleted sixth object in the second object corresponding to the fifth object relative to the second object corresponding to the fifth object; if the proportion meets a first threshold, delete the fifth object; if the proportion is greater than or equal to a second threshold, re-aggregate the seventh object in the second object corresponding to the fifth object except the deleted sixth object, wherein the second threshold is less than the first threshold.

[0089] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0090] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0091] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0092] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0093] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0094] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0095] An embodiment of the present application also provides a computer program, which includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in any one of the above method embodiments.

[0096] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0097] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0098] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. An object storage method, characterized in that: Applied to a storage device, the storage device includes a storage bucket, the storage bucket is respectively connected to a plurality of data pools and a classification pool, the classification pool is connected to the plurality of data pools, The method comprises: In a case where it is determined that the data size of the first object received through the storage bucket is less than a first value, storing the first object in the classification pool; When it is determined that the data capacity in the classification pool reaches a second value, the second object in the classification pool is stored in the multiple data pools according to the storage class corresponding to the second object, wherein the second object includes: the first object stored in the classification pool, and the second value is greater than the first value.

2. The method according to claim 1, characterized in that Before storing the first object in the classification pool, the method further includes: determining a data size of a first object received through the storage bucket; When the data size is greater than or equal to the first value, the first object is stored in the multiple data pools.

3. The method according to claim 1, characterized in that Storing the second object in the classification pool in the multiple data pools according to the storage class corresponding to the second object includes: determining whether the second object has a first storage class, wherein the first storage class is a storage class that is specified when the second object is sent to the storage device; In the case where the second object has the first storage class, storing the second object in the plurality of data pools through the first storage class; In a case where the second object does not have the first storage class, determining whether a first rule is enabled for the storage bucket, wherein the first rule is a matching rule for indicating that the second object is matched with the plurality of data pools; When the first rule is enabled in the storage bucket, storing the second object in the multiple data pools according to the first rule; If the first rule is not enabled for the storage bucket, determining whether a second rule is enabled for the storage bucket; When the second rule is enabled, the second object is stored in the multiple data pools based on the status information respectively corresponding to the multiple data pools according to the second rule.

4. The method according to claim 3, characterized in that Storing the second object in the plurality of data pools according to the first rule includes: Determine that the matching rule is a matching condition configured for any data pool, wherein the matching condition includes at least one of the following: an object prefix of a third object allowed to be stored in any data pool, an object size of the third object, and an object suffix of the third object; The second object satisfying the matching condition is stored in any one of the data pools.

5. The method according to claim 3, characterized in that: Storing the second object in the multiple data pools based on the state information respectively corresponding to the multiple data pools according to the second rule includes: Obtaining the status information respectively corresponding to the multiple data pools at the current moment, wherein the current moment is the moment when a fourth object needs to be stored from the classification pool to the multiple data pools, and the fourth object is any object among the second objects, wherein the status information includes: the creation time of each data pool, the remaining capacity of each data pool, and the remaining capacity ratio of the remaining capacity to the capacity of each data pool; The fourth object is stored in a target data pool whose status information satisfies the second rule, so as to store the second object in the multiple data pools, wherein the second rule includes at least one of the following: earliest creation time, largest remaining capacity, and highest remaining capacity ratio.

6. The method according to claim 1, characterized in that Storing the second object in the classification pool in the multiple data pools according to the storage class corresponding to the second object includes: In a case where it is determined that the storage bucket has enabled the third rule, aggregating the second object into a fifth object according to the third rule, wherein a third value corresponding to the data size of the fifth object is greater than the first value; The fifth object is stored in the plurality of data pools.

7. The method according to claim 6, characterized in that After aggregating the second objects into a fifth object according to the third rule, the method further includes: Determine a ratio of the deleted sixth object in the second object corresponding to the fifth object to the second object corresponding to the fifth object; When the proportion meets a first threshold, deleting the fifth object; When the proportion is greater than or equal to a second threshold, the seventh object in the second object corresponding to the fifth object except for the deleted sixth object is re-aggregated, wherein the second threshold is less than the first threshold.

8. An object storage device, characterized in that: Applied to a storage device, the storage device includes a storage bucket, the storage bucket is respectively connected to a plurality of data pools and a classification pool, the classification pool is connected to the plurality of data pools, The device comprises: a first storage module, configured to store the first object in the classification pool if it is determined that the data size of the first object received through the storage bucket is less than a first value; A second storage module is used to store the second object in the classification pool into the multiple data pools according to the storage class corresponding to the second object in the classification pool when it is determined that the data capacity in the classification pool has reached a second value, wherein the second object includes: the first object stored in the classification pool, and the second value is greater than the first value.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 7 when executed by a processor.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 7 are implemented.

Images