Object storage data storage management method, device and equipment

Abstract

The present application discloses an object storage data storage management method, device and equipment. When the reuse period is reached, a ZG cluster matching the data volume is allocated according to the estimated data volume of the source data file received in the reuse period; in response to the object write instruction of the source data file, a ZG is selected from the ZG clusters allocated in the current reuse period, and the storage node where the zone in the ZG is located is instructed to execute the writing of the object slice of the object; according to the life cycle of the source data file and the writing time of the object slice of the source data file, any expired ZG or ZG cluster is deleted. The problem of space waste and timely space recovery caused by the whole zone deletion feature that the SMR hard disk only supports continuous zone writing is solved, and the space utilization rate of the entire object storage system is improved.

Description

Technical Field

[0001] The present application relates to the field of cloud storage technology, and in particular to an object storage data storage management method, device and equipment. Background Art

[0002] Shingled magnetic recording (SMR) is a leading next-generation disk technology. Adjacent tracks are partially overlapped in sequence, which can increase the storage density of the unit storage medium and reduce storage costs. It is precisely because of the physical characteristics of SMR disks that its read behavior is the same as that of ordinary HDD (Hard Disk Drive) mechanical hard disks, but its write behavior has a huge difference. It does not support random writes and in-place update writes, because this will cause the overlapping track data to be overwritten. Therefore, SMR disks only support sequential writes from beginning to end.

[0003] SMR disks divide tracks into multiple bands, which are areas where continuous tracks are written to continuously. Each area becomes a basic unit that needs to be written sequentially. Band is a physical concept of SMR disks. The corresponding logical concept is called "zone". The size of a zone is 256MB. That is, the zone only supports continuous writing, random reading, and only supports the deletion of the entire zone space, resulting in space waste and the inability to recycle space in a timely manner.

[0004] Based on the above shortcomings of SMR, the current research focuses more on how to use SMR hard disks specifically, such as creating local file systems, creating SMR hard disk internal indexes, and the relationship between SMR and random storage media. None of them involve research on global management, space management, collection data types, data reliability, etc. at the distributed system level. It does not effectively solve the space waste and timely space recovery problems caused by the SMR hard disk only supporting continuous zone writing and the whole zone deletion feature, and does not perform cloud storage space management for global security streaming data. Summary of the invention

[0005] The present application provides a method, apparatus and device for managing storage space of object storage data, which are used for storage space management of object storage data and improve the space utilization rate of the entire cloud storage system based on SMR hard disks.

[0006] In a first aspect, an embodiment of the present application provides a method for managing object storage data storage space, the method comprising:

[0007] When the reuse period arrives, a ZG cluster matching the data volume is allocated according to the estimated data volume of the source data file received in the reuse period, wherein the ZG cluster includes multiple ZGs, and the number of zones in the ZG is consistent with the number N+M of object shards of the object of the source data file;

[0008] In response to the object write instruction of the source data file, a ZG is selected from the ZG cluster allocated in the current reuse period, and a storage node where the zone in the ZG is located is instructed to execute the writing of the object slice of the object;

[0009] According to the life cycle of the source data file and the writing time of the object slice of the source data file, any expired ZG or ZG cluster is deleted.

[0010] In a possible embodiment, selecting a ZG from a ZG cluster allocated in a current reuse period includes:

[0011] When determining that the current object is the first object in the reuse cycle, writing the current object into one of the ZGs in the ZG cluster;

[0012] When it is determined that the current object is not the first object in the reuse cycle, if the remaining space of the ZG where the previous object was written is available, the current object is written to the ZG where the previous object was written. If the remaining space of the ZG where the previous object was written is not available, an unused ZG is selected from the ZG cluster to write the object.

[0013] In a possible embodiment, the method further includes:

[0014] If the remaining space of the ZG where the previous object was written is not available and there is no unused ZG in the ZG cluster, according to the storage node load balancing strategy, N+M unused zones are reselected from the storage node to form a ZG for writing object shards.

[0015] In a possible embodiment, according to the estimated data volume of the source data file received within the multiplexing period, a ZG cluster matching the data volume is allocated, including:

[0016] Based on the information of all zones of the storage node, determine all unused zones and the storage nodes where they are located;

[0017] Determine the number n of ZGs to be allocated according to the estimated data volume of the source data file received within the multiplexing period;

[0018] According to the load balancing strategy between storage nodes, N+M unused zones are selected from different storage nodes to complete the allocation of one ZG, until the allocation of n ZGs is completed.

[0019] In a possible embodiment, according to the estimated data volume of the source data file received within the multiplexing period, a ZG cluster matching the data volume is allocated, including:

[0020] Based on the information of all zones of the storage node, determine all unused zones and the storage nodes where they are located;

[0021] Determine the number n of ZGs to be allocated according to the estimated data volume of the source data file received within the multiplexing period;

[0022] For each ZG to be allocated, determine the corresponding N+M storage nodes according to the load balancing strategy between storage nodes;

[0023] In response to the object write instruction of the source data file, selecting a ZG from the ZG clusters allocated in the current reuse cycle, comprising:

[0024] In response to the object write instruction of the source data file, a ZG is selected from the ZG cluster allocated in the current reuse period, the N+M storage nodes corresponding to the selected ZG are determined, and according to the internal load balancing strategy of the storage nodes, an unused zone is selected from the N+M storage nodes respectively.

[0025] In a possible embodiment, in response to the object write instruction of the source data file, the method further includes:

[0026] Determine that the current object is data written for the first time after a previous stream interruption occurs, and when it is determined that the interruption time does not exceed a preset time, obtain the first ZG to which the last object was written before the interruption occurs, and when it is determined that the remaining space of the first ZG is available, write the current object into the first ZG;

[0027] Otherwise, an unused ZG is selected from the ZG cluster to write the object.

[0028] In a possible embodiment, after instructing the storage node where the zone in the ZG is located to execute writing of the object slice of the object, the method further includes:

[0029] Establishing a bidirectional index relationship between the object and the ZG cluster allocated in the current cycle and the selected ZG; or

[0030] The time when the object is written into the selected ZG is recorded, and the ZGs in the ZG cluster are sorted according to the time when the first object in the ZG is written.

[0031] In a possible embodiment, the method further includes:

[0032] Receive a file reading request sent by a client, wherein the file reading request includes a file identifier and a collection time of a source data file to be read;

[0033] According to the acquisition time of the source data file, searching for the file information of all the source data files corresponding to the acquisition time, and searching for the file information of the source data file to be read according to the file identifier in the found file information of the source data file;

[0034] According to the found file information of the source data file to be read, determine the objects included in the source data file to be read, the ZG cluster to which the objects belong, the starting ZG of the objects in the ZG cluster to which they belong and their offset in the ZG, the ending ZG and their offset and length in the ending ZG;

[0035] From the ZG cluster to which the object belongs, reading starts from the starting position determined by the starting ZG of the object in the ZG cluster and the offset in the ZG, and ends at the ending position determined by the ending ZG and the offset and length in the ending ZG.

[0036] In a possible embodiment, according to the life cycle of the source data file and the writing time of the object slice of the source data file, deleting any expired ZG or ZG cluster includes:

[0037] According to the expiration time of the ZG cluster, determine and delete the expired ZG cluster, where the expiration time of the ZG cluster is the time obtained by summing the first writing time of the first object in the ZG of the first object written in the ZG cluster, the life cycle of the source data file and the reuse period of the ZG cluster;

[0038] According to the expiration time of ZG, the expired ZG is determined and deleted. The expiration time of ZG is the time obtained by summing the second write time of the first object in the ZG, the life cycle of the source data file and the reuse period of the ZG.

[0039] In a possible embodiment, at least one of the following steps is also included:

[0040] When it is determined that the life cycle of the source data file has changed or the EC mode of the object of the source data file has changed, unused ZGs in the ZG cluster allocated in the current reset period are released, and reallocation of the ZG cluster is triggered;

[0041] When it is determined that the space of the allocated ZG cluster cannot meet the storage demand before the end of the current service cycle, the next reuse cycle is triggered to re-allocate the ZG cluster;

[0042] When it is determined that there is an unused ZG in the ZG cluster at the end of the current reuse period, the unused ZG is released from the ZG cluster.

[0043] In a possible embodiment, the method further includes:

[0044] It is detected that there is a second ZG with an abnormal zone in the ZG cluster of the current reuse period;

[0045] When it is determined that the second ZG is not in use, reselect a new ZG to replace the second ZG;

[0046] When it is determined that the second ZG has been used, data is restored for the second ZG when it is determined that a recovery condition is met based on the number of zones in the second ZG where no data loss occurs.

[0047] In a possible embodiment, in response to the ZG integrity detection instruction, it is detected that there is a second ZG with an abnormal zone in all the allocated ZG clusters;

[0048] When it is determined that the recovery condition is met according to the number of zones in the second ZG where no data loss occurs, data recovery is performed on the second ZG.

[0049] In a second aspect, an embodiment of the present application provides an object storage data storage space management device, the device comprising:

[0050] A ZG cluster pre-allocation module is used to allocate a ZG cluster matching the data volume according to the estimated data volume of the source data file received within the reuse period, wherein the ZG cluster includes multiple ZGs, and the number of zones in the ZG is consistent with the number N+M of object shards of the object of the source data file;

[0051] A data writing module, configured to respond to an object writing instruction of the source data file, select a ZG from the ZG cluster allocated in the current reuse period, and instruct the storage node where the zone in the ZG is located to execute writing of the object slice of the object;

[0052] The ZG cluster deletion module is used to delete any expired ZG or ZG cluster according to the life cycle of the source data file and the writing time of the object segment of the source data file.

[0053] In a third aspect, an embodiment of the present application provides an object storage data storage space management device, the device comprising:

[0054] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the methods described in the object storage data storage space management method.

[0055] In a fourth aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a computer program, and the computer program is used to enable a computer to execute any one of the methods for managing object storage data storage space.

[0056] The embodiments of the present application provide an object storage data storage space management method, device and equipment, which achieve the effect of efficient recycling of cloud storage space by applying for, managing and dynamically adjusting global space for data storage and recovering global space for data deletion, solve the problems of space waste and timely space recovery caused by the SMR hard disk only supporting continuous zone writing and the whole zone deletion feature, and improve the space utilization of the entire object storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A schematic diagram of a ZG composition according to an exemplary embodiment of the present invention;

[0058] Figure 2 A schematic diagram of a flow chart of a method for managing object storage data storage space according to an exemplary embodiment of the present invention;

[0059] Figure 3 A schematic diagram of a data writing process according to an exemplary embodiment of the present invention;

[0060] Figure 4 A schematic diagram of an object writing process according to an exemplary embodiment of the present invention;

[0061] Figure 5 A schematic diagram of an object storage data storage space management device according to an exemplary embodiment of the present invention;

[0062] Figure 6 The present invention is a schematic diagram of an object storage data storage space management device according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0063] The technical solutions in the embodiments of the present application will be described clearly and in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] First, the professional terms of the embodiments of the present invention are introduced.

[0065] Shingled magnetic recording (SMR) disk: is a leading next-generation disk technology. Adjacent tracks are partially overlapped in sequence, which can increase the storage density of the unit storage medium and reduce storage costs. It is precisely because of the physical characteristics of SMR disks that its read behavior is the same as that of ordinary mechanical hard disks (Hard Disk Drives, HDDs), but its write behavior has a huge change. It does not support random writes and in-place update writes, because this will cause the overlapping track data to be overwritten. Therefore, SMR disks only support sequential writes from beginning to end.

[0066] SMR disks divide tracks into multiple bands, which are areas where continuous tracks are written to continuously. Each area becomes a basic unit that needs to be written sequentially. Band is a physical concept of SMR disks, and the corresponding logical concept is called "zone". The size of a zone is 256MB. That is, the zone only supports continuous writing, supports random reading, and only supports deletion of the entire zone space.

[0067] Erasure Code (EC): The general term is N+M, that is, N object shards generate M check blocks.

[0068] Data lifecycle: refers to the effective storage time of data. For example, in security storage scenarios, the video and image storage cycles are usually measured in "hours". For example, video storage is 90 days * 24 hours, and image storage is 180 days * 24 hours. After the data lifecycle expires, the system automatically deletes the expired data, frees up space for new data writing, and recycles the storage space.

[0069] Zone Group (ZG): A collection of zones. N+M zones are selected from N+M SMR hard disks (one zone is selected from each hard disk) to form a ZG. In a distributed storage system, considering the reliability of data storage, the zones in a ZG usually come from different racks, different storage nodes, and different SMR hard disks. The number of N+M is the same as that in the EC mode.

[0070] like Figure 1 As shown, taking 4+1 as an example, ZG consists of zone1, zone2, zone3, zone4, and zone5, and each zone comes from a different storage node.

[0071] The object storage data storage space management method in the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0072] Figure 2A schematic diagram of a process flow of an object storage data storage space management method provided in an embodiment of the present application, application and management node, including:

[0073] S201: When a reuse period is reached, a ZG cluster matching the data volume is allocated according to the estimated data volume of the source data file received within the reuse period, wherein the ZG cluster includes multiple ZGs, and the number of zones in the ZG is consistent with the number N+M of object shards of the object of the source data file.

[0074] The user writes data into the cloud storage system through the client, and a bucket directory corresponding to the customer will be established in the storage system. The source data file in the embodiment of the present application is written into the cloud storage system in accordance with the EC mode, that is, a data file can be divided into multiple objects, and the object is sharded according to the preset object shard size, and then multiple redundant object shards are calculated, and finally N+M object shards are obtained.

[0075] Within the preset reuse period, the data volume of the source data file uploaded to the cloud storage system can be estimated. According to the estimated data volume, a ZG cluster matching the data volume is allocated. The ZG cluster includes multiple ZGs, and the ZG includes N+M zones, where the N+M zones can come from the same storage node. However, out of consideration for data reliability, the N+M zones included in the ZG in the embodiment of the present application come from different storage nodes, where the N+M zones are all idle zones. The reuse period is the period for allocating ZG clusters, which can be configured according to user needs.

[0076] The management node requires the storage nodes to report all zone information on all storage nodes, including the number of used zones and unused zones, the used zones include the zones that have been written but not full and the zones that are full; the number of allocated zones and the capacity information of each storage node. The above information is obtained by the management node instructing each storage node to report and / or each storage node reporting according to the preset time granularity.

[0077] S202: In response to the object write instruction of the source data file, a ZG is selected from the ZG cluster allocated in the current reuse period, and the storage node where the zone in the ZG is located is instructed to execute the writing of the object slice of the object.

[0078] Selecting a ZG from the ZG clusters allocated in the current reuse cycle requires selection based on the write order of the current object, the remaining ZG space written by the previous object, the capacity of the current ZG cluster, and the interruption time of the data stream.

[0079] It should be noted that the writing process in the embodiments of the present application is aligned writing, that is, N+M objects are written to the N+M zones of the current ZG at the same time.

[0080] S203: Deleting any expired ZG or ZG cluster according to the life cycle of the source data file and the writing time of the object slice of the source data file.

[0081] The source data files of the same client belong to the same Bucket, and the lifecycle of the source data files is managed at the Bucket granularity. List all source data file information under the Bucket owned by the client, and the source data file information includes the object segment write time of each ZG in the ZG cluster allocated to the source data file. If the current system time is beyond the lifecycle from the time when the object segment is written, the ZG or ZG cluster deletion condition is met, the ZG or ZG cluster is deleted, and the corresponding storage node is instructed to recycle the idle zone and reset the write pointer, and the available resources of the idle zone are uniformly managed. The deletion method described above does not cause delayed release of the zone due to zone data holes, which greatly improves the system space utilization.

[0082] An object storage data storage space management method provided by an embodiment of the present application is based on an SMR hard disk on a storage node. By logically allocating ZG space in advance, data from the same data source is written into the pre-allocated ZG. After the data lifecycle expires, the entire ZG space is quickly reclaimed. That is, the application and release of data write space is logically the application and release of the entire ZG block. It can also flexibly adapt to streaming data source writing, support frequent lifecycle changes, dynamically adapt to EC write mode changes and other scenarios, so as to achieve the purpose of improving cloud storage space utilization.

[0083] In the above step S201, the corresponding relationship between the data file and ZG is as follows: Figure 3 As shown, the user writes data into the cloud storage system through the client in accordance with the EC mode. The allocated ZG cluster is constructed based on the time strategy of ZG allocation. For example, the data volume of one day is estimated, and 8 ZGs are allocated for one day's data. The 8 ZGs are managed uniformly to obtain the ZG cluster allocated within one day. The time can be configured according to the user's wishes. The zones included in the same ZG in the ZG cluster come from different storage nodes, and the zones in different ZGs can come from the same storage node.

[0084] In a possible implementation manner, according to the estimated data volume of the source data file received within the multiplexing period, allocating a ZG cluster matching the data volume includes at least one of the following situations:

[0085] As a possible implementation, in the process of allocating a ZG cluster matching the estimated data volume of the source data file received within the multiplexing period, at least one of the following situations is included:

[0086] 1) The management node only records the number of zones that the ZG cluster needs to allocate on each storage node. Before writing data, the offset address of a specific SMR disk of the storage node and the corresponding specific zone is obtained.

[0087] Based on the information of all zones of the storage node, determine all unused zones and the storage nodes where they are located;

[0088] Determine the number n of ZGs to be allocated according to the estimated data volume of the source data file received within the multiplexing period;

[0089] For each ZG to be allocated, determine the corresponding N+M storage nodes according to the load balancing strategy between storage nodes;

[0090] In response to the object write instruction of the source data file, selecting a ZG from the ZG clusters allocated in the current reuse cycle, comprising:

[0091] In response to the object write instruction of the source data file, a ZG is selected from the ZG cluster allocated in the current reuse period, the N+M storage nodes corresponding to the selected ZG are determined, and according to the internal load balancing strategy of the storage nodes, an unused zone is selected from the N+M storage nodes respectively.

[0092] 2) The management node records the number of zones that the ZG cluster needs to allocate on each storage node, as well as the offset address of a specific SMR disk of the storage node and the corresponding specific zone.

[0093] Based on the information of all zones of the storage node, determine all unused zones and the storage nodes where they are located;

[0094] Determine the number n of ZGs to be allocated according to the estimated data volume of the source data file received within the multiplexing period;

[0095] According to the load balancing strategy between storage nodes, N+M unused zones are selected from different storage nodes to complete the allocation of one ZG, until the allocation of n ZGs is completed.

[0096] 3) It can be allocated according to the reuse period.

[0097] For example, based on the estimated data volume of 1 day being 8GB, taking the 4+1 mode as an example, first divide the 8GB data file into 8 1GB objects, and allocate ZG space to the 8 1GB objects according to the 4+1 mode. Then, each 1GB object will be allocated to one ZG, and 8GB of data will be allocated to 8 ZGs. Each ZG includes 5 zones, and each zone comes from a different storage node.

[0098] If the data life cycle is 30 days, the data volume for 30 days can be estimated and the required ZGs can be allocated for the data for 30 days at one time. The above allocated ZGs include multiple idle zones, and the reuse period is the data life cycle.

[0099] 4) When it is determined that the life cycle of the source data file has changed or the EC mode of the object of the source data file has changed, unused ZGs in the ZG cluster allocated in the current reset cycle are released, and reallocation of the ZG cluster is triggered.

[0100] EC mode change is N+M change, for example, from 4+1 mode to 4+2 mode. In this case, the number of zones in ZG will also change, so the ZGs in the previously pre-allocated ZG cluster cannot be used anymore. The current ZG cluster will release the unused ZGs and then rebuild the ZG cluster according to the changed 4+2 mode.

[0101] 5) When it is determined that the space of the allocated ZG cluster cannot meet the storage demand before the end of the current service cycle, the ZG cluster allocation is triggered to be re-allocated in the next reuse cycle.

[0102] When the data stream specification is increased or the data volume estimation error causes the remaining space in the ZG cluster allocated in the current reuse period to fail to meet the storage demand (for example, the remaining space is less than 5% of the total space, 5% is configurable), the reallocation of the ZG cluster for the next reuse period is triggered in advance.

[0103] The above expansion of the unused ZG is real-time and dynamic, and does not require stopping the current data writing action, that is, the expansion is performed without the user's awareness.

[0104] According to the above allocation method, each time data is written, the most reasonable hard disk and the most reasonable zone are selected, thereby maximizing the stability of the entire system.

[0105] The above step S202 can be specifically implemented as follows:

[0106] During the writing process, data is written continuously, that is, the object is first written into a ZG. When the ZG is full or the remaining space is insufficient to write the next object, another ZG is selected from the current ZG cluster and the unwritten object is written into the ZG.

[0107] In a possible implementation, selecting a ZG from a ZG cluster allocated in a current reuse period includes:

[0108] When it is determined that the current object is the first object in the reuse cycle, the current object is written into one of the ZGs in the ZG cluster. If it is the first object, it means that all ZGs in the ZG cluster are available, and any ZG can be selected.

[0109] When it is determined that the current object is not the first object in the reuse cycle, if the remaining space of the ZG where the previous object was written is available, the current object will be written to the ZG where the previous object was written. If the remaining space of the ZG where the previous object was written is not available, an unused ZG will be selected from the ZG cluster to write the object. This will ensure that when a ZG is full or the remaining space is unavailable, the next ZG will be written to ensure that objects with similar time are written to the same ZG.

[0110] In a possible implementation, if the remaining space of the ZG to which the previous object was written is not available, and there is no unused ZG in the ZG cluster, it means that the ZG allocated in the current reuse cycle is insufficient. Then, according to the storage node load balancing strategy, N+M unused zones are reselected from the storage node to form a ZG for writing object shards, and the reselected ZG is expanded to the current ZG cluster.

[0111] During the data writing process, an interruption may have occurred before. In this embodiment, it is determined that the current object is the data written for the first time after the previous stream interruption. When it is determined that the interruption time does not exceed the preset time, the first ZG to which the last object was written before the interruption occurs is obtained. When it is determined that the remaining space of the first ZG is available, the current object is written into the first ZG.

[0112] Otherwise, an unused ZG is selected from the ZG cluster to write the object.

[0113] In this way, after a period of data interruption, the new object will be written to the ZG that was not filled last time. In the embodiment of the present application, the interruption time usually does not exceed half of the data life cycle. For example, if the life cycle is 180 days, the data writing time in the ZG should not differ by 90 days. The above-mentioned reuse time in this embodiment is a configurable item.

[0114] The specific data writing process is as follows Figure 4 As shown:

[0115] Data from the same data source is written to the same ZG cluster; for streaming data from the same client that is close in time, even if the streaming data writing is interrupted (there is an interruption time), after the streaming data is restored, it will continue to be written to the last unfull ZG. That is, data that is close in time is written to the same ZG.

[0116] Step 1: First receive the object fragment sent from the client, and determine whether the current object is the first data written after the previous stream interruption. If so, obtain the first ZG written by the last object before the interruption. If not, execute step 3.

[0117] Step 2: Determine whether the interruption time exceeds the preset time. If not, determine whether the remaining space of the ZG is available. If it exceeds, execute step 3; if the remaining space is available, obtain the writable offset address of the ZG, establish the correspondence between the file and the ZG and the ZG cluster to write data, if the remaining space is not available, execute step 3.

[0118] Step 3: Get any unused ZG from the current ZG cluster, establish the correspondence between the file, ZG and ZG cluster to write data.

[0119] Step 4: After the object file is written to a certain size, switch to writing a new object file and establish the corresponding relationship between the new object file and ZG and ZG cluster.

[0120] It should be noted that, in order to ensure the persistence of information, after instructing the storage node where the zone in the ZG is located to write the object slice of the object, a bidirectional index relationship between the object and the ZG cluster allocated in the current cycle and the selected ZG is established; or

[0121] The time when the object is written into the selected ZG is recorded, and the ZGs in the ZG cluster are sorted according to the time when the first object in the ZG is written.

[0122] In a possible implementation, during the data writing process, the following situation also exists:

[0123] 1) When it is determined that there is an unused ZG in the ZG cluster at the end of the current reuse period, the unused ZG is released from the ZG cluster.

[0124] When the specification of the data code stream is reduced or due to an error in estimating the data volume, resulting in unused ZGs remaining in the ZG cluster allocated in the current multiplexing period, the unused ZGs are released from the ZG cluster.

[0125] 2) When it is detected that there is a second ZG with an abnormal zone in the ZG cluster of the current reuse period and it is determined that the second ZG is not in use, a new ZG is reselected to replace the second ZG.

[0126] There are many cases where the zone is abnormal, such as failure of the rack, storage node, or SMR hard disk. In the above cases, if the current ZG is not in use, a new ZG is selected to replace the current ZG. The selection process is as described above.

[0127] Abnormal zones include those caused by disk removal, bad disk, bad sectors, node deletion, node offline and timeout. The management node will reset the corresponding zone information in the ZG cache (the zone length and the storage node position are reset to 0), and the zone whose object fragment length or write pointer position in the first ZG is reset to 0 is determined to be an abnormal zone.

[0128] 3) It is detected that there is a second ZG with an abnormal zone in the ZG cluster of the current reuse period and it is determined that the second ZG has been used. When it is determined that the recovery condition is met, data recovery is performed on the second ZG.

[0129] In response to the ZG integrity detection instruction, it is detected that there is a second ZG with an abnormal zone in all the allocated ZG clusters;

[0130] When it is determined that the recovery condition is met according to the number of zones in the second ZG where no data loss occurs, data recovery is performed on the second ZG.

[0131] Perform data length statistics on each zone in the second ZG where the abnormal zone appears, obtain at least one object fragment length, and determine the number of zones corresponding to each object fragment length;

[0132] When the maximum number of zones L meets the recovery condition, it is determined that the ZG to be recovered can be recovered. The recovery condition is: L satisfies N≤L <N+M。

[0133] The above step S203 can be specifically implemented as follows:

[0134] In a possible implementation manner, deleting any expired ZG or ZG cluster specifically includes at least one of the following situations:

[0135] 1) Deletion is performed at the granularity of ZG clusters.

[0136] According to the expiration time of the ZG cluster, the expired ZG cluster is determined and deleted. The expiration time of the ZG cluster is the time obtained by summing the first write time of the first object in the ZG that first writes the object in the ZG cluster, the life cycle of the source data file and the reuse period of the ZG cluster.

[0137] For example, a ZG cluster is managed with a granularity of 1 day, that is, the reuse period of the ZG cluster is 1 day, and the life cycle of the source data file is 30 days. If the write time of the first object in the ZG that first writes the object in the ZG cluster exceeds 31 (30+1) days, the ZG cluster is determined to be expired, all ZGs in the ZG cluster are released, all zones corresponding to all ZGs are released, and the management node instructs the storage node to recycle the zone and reset the write pointer, and put it into the available zone space pool for unified management of available resources. After the ZG cluster is successfully deleted, all associated object file record information corresponding to the ZG cluster is deleted.

[0138] 2) Delete with ZG as the granularity.

[0139] According to the expiration time of ZG, the expired ZG is determined and deleted. The expiration time of ZG is the time obtained by summing the second write time of the first object in the ZG, the life cycle of the source data file and the reuse period of the ZG.

[0140] For example, if the life cycle of a source data file is 30 days and the reuse period of the ZG is 1 day, if the write time of the first object in the ZG exceeds 31 (30+1) days, the ZG is determined to be expired, and all zones corresponding to all ZGs are released, and all file records associated with the ZG are deleted.

[0141] In particular, if the object file spans multiple ZGs, the file is marked as "invalid, to be deleted".

[0142] The physical granularity of the space released in the above two methods is the size of the zone space of the SMR hard disk. The whole block is deleted and reset. There is no delayed zone release caused by zone data holes. The system space utilization rate is greatly improved.

[0143] In a possible implementation, the process of reading data from the cloud storage system is as follows:

[0144] Receive a file reading request sent by a client, wherein the file reading request includes a file identifier and a collection time of a source data file to be read;

[0145] According to the acquisition time of the source data file, searching for the file information of all the source data files corresponding to the acquisition time, and searching for the file information of the source data file to be read according to the file identifier in the found file information of the source data file;

[0146] According to the found file information of the source data file to be read, determine the objects included in the source data file to be read, the ZG cluster to which the objects belong, the starting ZG of the objects in the ZG cluster to which they belong and their offset in the ZG, the ending ZG and their offset and length in the ending ZG;

[0147] From the ZG cluster to which the object belongs, reading starts from the starting position determined by the starting ZG of the object in the ZG cluster and the offset in the ZG, and ends at the ending position determined by the ending ZG and the offset and length in the ending ZG.

[0148] For example, if a user wants to read the video recorded from 9:00 to 10:00 on September 1, the user finds the corresponding Bucket directory based on the information of the client, including the file identifier and the acquisition time, and lists all the files under the Bucket directory. There may be multiple files from 9:00 to 10:00. The files from 9:00 to 10:00 are searched in all the files, and then the corresponding objects, the ZG cluster to which the objects belong, the starting ZG of the object in the ZG cluster and its offset in ZG, the ending ZG and its offset and length in the ending ZG can be determined. Reading starts from the starting position determined by the starting ZG of the object in the ZG cluster and its offset in ZG, and reads in sequence according to the chronological order in which the objects are written to ZG.

[0149] Based on the same inventive concept, the present application provides an object storage data lifecycle management device, 500, such as Figure 5 As shown, the device comprises:

[0150] A ZG cluster pre-allocation module 501 is used to allocate a ZG cluster matching the data volume according to the estimated data volume of the source data file received within the reuse period, wherein the ZG cluster includes multiple ZGs, and the number of zones in the ZG is consistent with the number N+M of object shards of the object of the source data file;

[0151] The data writing module 502 is used to select a ZG from the ZG cluster allocated in the current reuse period in response to the object writing instruction of the source data file, and instruct the storage node where the zone in the ZG is located to execute the writing of the object slice of the object;

[0152] The ZG cluster deletion module 503 is used to delete any expired ZG or ZG cluster according to the life cycle of the source data file and the writing time of the object slice of the source data file.

[0153] In a possible implementation, the ZG cluster pre-allocation module selects a ZG from the ZG clusters allocated in the current reuse period, including:

[0154] When determining that the current object is the first object in the reuse cycle, writing the current object into one of the ZGs in the ZG cluster;

[0155] When it is determined that the current object is not the first object in the reuse cycle, if the remaining space of the ZG where the previous object was written is available, the current object is written to the ZG where the previous object was written. If the remaining space of the ZG where the previous object was written is not available, an unused ZG is selected from the ZG cluster to write the object.

[0156] In a possible implementation manner, the ZG cluster pre-allocation module is further used for:

[0157] If the remaining space of the ZG where the previous object was written is not available and there is no unused ZG in the ZG cluster, according to the storage node load balancing strategy, N+M unused zones are reselected from the storage node to form a ZG for writing object shards.

[0158] In a possible implementation, the ZG cluster pre-allocation module allocates a ZG cluster matching the data amount according to the estimated data amount of the source data file received within the reuse period, including:

[0159] Based on the information of all zones of the storage node, determine all unused zones and the storage nodes where they are located;

[0160] Determine the number n of ZGs to be allocated according to the estimated data volume of the source data file received within the multiplexing period;

[0161] According to the load balancing strategy between storage nodes, N+M unused zones are selected from different storage nodes to complete the allocation of one ZG, until the allocation of n ZGs is completed.

[0162] In a possible implementation, the ZG cluster pre-allocation module allocates a ZG cluster matching the data amount according to the estimated data amount of the source data file received within the reuse period, including:

[0163] Based on the information of all zones of the storage node, determine all unused zones and the storage nodes where they are located;

[0164] Determine the number n of ZGs to be allocated according to the estimated data volume of the source data file received within the multiplexing period;

[0165] For each ZG to be allocated, determine the corresponding N+M storage nodes according to the load balancing strategy between storage nodes;

[0166] In response to the object write instruction of the source data file, selecting a ZG from the ZG clusters allocated in the current reuse cycle, comprising:

[0167] In response to the object write instruction of the source data file, a ZG is selected from the ZG cluster allocated in the current reuse period, the N+M storage nodes corresponding to the selected ZG are determined, and according to the internal load balancing strategy of the storage nodes, an unused zone is selected from the N+M storage nodes respectively.

[0168] In a possible implementation manner, the data writing module responds to the object writing instruction of the source data file and further includes:

[0169] Determine that the current object is data written for the first time after a previous stream interruption occurs, and when it is determined that the interruption time does not exceed a preset time, obtain the first ZG to which the last object was written before the interruption occurs, and when it is determined that the remaining space of the first ZG is available, write the current object into the first ZG;

[0170] Otherwise, an unused ZG is selected from the ZG cluster to write the object.

[0171] In a possible implementation manner, after the data writing module instructs the storage node where the zone in the ZG is located to execute writing of the object slice of the object, the method further includes:

[0172] Establishing a bidirectional index relationship between the object and the ZG cluster allocated in the current cycle and the selected ZG; or

[0173] The time when the object is written into the selected ZG is recorded, and the ZGs in the ZG cluster are sorted according to the time when the first object in the ZG is written.

[0174] In a possible implementation, the data reading module receives a file reading request sent by a client, wherein the file reading request includes a file identifier and a collection time of a source data file to be read;

[0175] According to the acquisition time of the source data file, searching for the file information of all the source data files corresponding to the acquisition time, and searching for the file information of the source data file to be read according to the file identifier in the found file information of the source data file;

[0176] According to the found file information of the source data file to be read, determine the objects included in the source data file to be read, the ZG cluster to which the objects belong, the starting ZG of the objects in the ZG cluster to which they belong and their offset in the ZG, the ending ZG and their offset and length in the ending ZG;

[0177] From the ZG cluster to which the object belongs, reading starts from the starting position determined by the starting ZG of the object in the ZG cluster and the offset in the ZG, and ends at the ending position determined by the ending ZG and the offset and length in the ending ZG.

[0178] In a possible implementation, the ZG cluster deletion module deletes any expired ZG or ZG cluster according to the life cycle of the source data file and the writing time of the object slice of the source data file, including:

[0179] According to the expiration time of the ZG cluster, determine and delete the expired ZG cluster, where the expiration time of the ZG cluster is the time obtained by summing the first writing time of the first object in the ZG of the first object written in the ZG cluster, the life cycle of the source data file and the reuse period of the ZG cluster;

[0180] According to the expiration time of ZG, the expired ZG is determined and deleted. The expiration time of ZG is the time obtained by summing the second write time of the first object in the ZG, the life cycle of the source data file and the reuse period of the ZG.

[0181] In a possible implementation manner, the ZG cluster pre-allocation module further includes at least one of the following steps:

[0182] When it is determined that the life cycle of the source data file has changed or the EC mode of the object of the source data file has changed, unused ZGs in the ZG cluster allocated in the current reset period are released, and reallocation of the ZG cluster is triggered;

[0183] When it is determined that the space of the allocated ZG cluster cannot meet the storage demand before the end of the current service cycle, the next reuse cycle is triggered to re-allocate the ZG cluster;

[0184] When it is determined that there is an unused ZG in the ZG cluster at the end of the current reuse period, the unused ZG is released from the ZG cluster.

[0185] In a possible implementation, the method of the ZG cluster pre-allocation module further includes:

[0186] It is detected that there is a second ZG with an abnormal zone in the ZG cluster of the current reuse period;

[0187] When it is determined that the second ZG is not in use, reselect a new ZG to replace the second ZG;

[0188] When it is determined that the second ZG has been used, data is restored for the second ZG when it is determined that a recovery condition is met based on the number of zones in the second ZG where no data loss occurs.

[0189] In a possible implementation, the ZG cluster pre-allocation module detects, in response to the ZG integrity detection instruction, that there is a second ZG with an abnormal zone in all allocated ZG clusters;

[0190] When it is determined that the recovery condition is met according to the number of zones in the second ZG where no data loss occurs, data recovery is performed on the second ZG.

[0191] Based on the same inventive concept, the present application provides an object storage data storage space management method and device, the device comprising:

[0192] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method as described in any one of the object storage data storage space management methods in the embodiments of the present application.

[0193] Refer to the following Figure 6 The electronic device 130 according to this embodiment of the present application is described. Figure 6 The electronic device 130 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0194] like Figure 6 As shown, the electronic device 130 is in the form of a general electronic device. The components of the electronic device 130 may include but are not limited to: the at least one processor 131, the at least one memory 132, and a bus 133 connecting different system components (including the memory 132 and the processor 131).

[0195] The processor 131 is used to read and execute instructions in the memory 132, so that the at least one processor can execute the object storage data storage space management method provided in the above embodiment.

[0196] Bus 133 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0197] The memory 132 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1321 and / or a cache memory 1322 , and may further include a read-only memory (ROM) 1323 .

[0198] The memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, such program modules 1324 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0199] The electronic device 130 may also communicate with one or more external devices 134 (e.g., keyboards, pointing devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 130, and / or communicate with any device that enables the electronic device 130 to communicate with one or more other electronic devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 135. Furthermore, the electronic device 130 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 136. As shown, the network adapter 136 communicates with other modules for the electronic device 130 via a bus 133. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0200] In some possible implementations, various aspects of an object storage data storage space management method provided by the present application may also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of an object storage data storage space management method according to various exemplary implementations of the present application described above in this specification.

[0201] In addition, the present application also provides a computer-readable storage medium, wherein the computer storage medium stores a computer program, wherein the computer program is used to enable a computer to execute any of the methods described in the above embodiments.

[0202] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0203] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0204] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0205] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for managing object storage data storage space, characterized in that: The method comprises: When the reuse period arrives, a ZG cluster matching the data volume is allocated according to the estimated data volume of the source data file received in the reuse period, wherein the ZG cluster includes multiple ZGs, and the number of zones in the ZG is consistent with the number N+M of object shards of the object of the source data file; In response to the object write instruction of the source data file, when it is determined that the current object is the first object in the reuse cycle, the current object is written into one of the ZGs in the ZG cluster; or when it is determined that the current object is not the first object in the reuse cycle, if the remaining space of the ZG into which the previous object was written is available, the current object is written into the ZG into which the previous object was written, and if the remaining space of the ZG into which the previous object was written is not available, an unused ZG is selected from the ZG cluster for object writing again; or if the remaining space of the ZG into which the previous object was written is not available, and there is no unused ZG in the ZG cluster, according to the storage node load balancing strategy, N+M unused zones are selected from the storage node to form a ZG for writing object fragments; or when it is determined that the current object is the data written for the first time after the previous stream interruption occurs, and when it is determined that the interruption time does not exceed the preset time, the first ZG into which the last object was written before the interruption occurs is obtained, and when it is determined that the remaining space of the first ZG is available, the current object is written into the first ZG, otherwise, an unused ZG is selected from the ZG cluster for object writing again; According to the life cycle of the source data file and the writing time of the object slice of the source data file, any expired ZG or ZG cluster is deleted.

2. The method according to claim 1, characterized in that: According to the estimated data volume of the source data file received within the multiplexing period, a ZG cluster matching the data volume is allocated, including: Based on the information of all zones of the storage node, determine all unused zones and the storage nodes where they are located; Determine the number n of ZGs to be allocated according to the estimated data volume of the source data file received within the multiplexing period; According to the load balancing strategy between storage nodes, N+M unused zones are selected from different storage nodes to complete the allocation of one ZG, until the allocation of n ZGs is completed.

3. The method according to claim 1, characterized in that According to the estimated data volume of the source data file received within the multiplexing period, a ZG cluster matching the data volume is allocated, including: Based on the information of all zones of the storage node, determine all unused zones and the storage nodes where they are located; Determine the number n of ZGs to be allocated according to the estimated data volume of the source data file received within the multiplexing period; For each ZG to be allocated, determine the corresponding N+M storage nodes according to the load balancing strategy between storage nodes; In response to the object write instruction of the source data file, a ZG is selected from the ZG cluster allocated in the current reuse period, the N+M storage nodes corresponding to the selected ZG are determined, and according to the internal load balancing strategy of the storage nodes, an unused zone is selected from the N+M storage nodes respectively.

4. The method according to claim 1, characterized in that After writing the object, it also includes: Establishing a bidirectional index relationship between the object and the ZG cluster allocated in the current cycle and the selected ZG; or The time when the object is written into the selected ZG is recorded, and the ZGs in the ZG cluster are sorted according to the time when the first object in the ZG is written.

5. The method according to claim 1, characterized in that Also includes: Receive a file reading request sent by a client, wherein the file reading request includes a file identifier and a collection time of a source data file to be read; According to the acquisition time of the source data file, searching for the file information of all the source data files corresponding to the acquisition time, and searching for the file information of the source data file to be read according to the file identifier in the found file information of the source data file; According to the found file information of the source data file to be read, determine the objects included in the source data file to be read, the ZG cluster to which the objects belong, the starting ZG of the objects in the ZG cluster to which they belong and their offset in the ZG, the ending ZG and their offset and length in the ending ZG; From the ZG cluster to which the object belongs, read from the starting position determined by the starting ZG of the object in the ZG cluster and the offset in the ZG, and end at the ending position determined by the ending ZG and the offset and length in the ending ZG.

6. The method according to claim 1, characterized in that According to the life cycle of the source data file and the writing time of the object slice of the source data file, deleting any expired ZG or ZG cluster includes: According to the expiration time of the ZG cluster, determine and delete the expired ZG cluster, where the expiration time of the ZG cluster is the time obtained by summing the first writing time of the first object in the ZG of the first object written in the ZG cluster, the life cycle of the source data file and the reuse period of the ZG cluster; According to the expiration time of ZG, the expired ZG is determined and deleted. The expiration time of ZG is the time obtained by summing the second write time of the first object in the ZG, the life cycle of the source data file and the reuse period of the ZG.

7. The method according to claim 1, characterized in that It also includes at least one of the following steps: When it is determined that the life cycle of the source data file has changed or the EC mode of the object of the source data file has changed, unused ZGs in the ZG cluster allocated in the current reset period are released, and reallocation of the ZG cluster is triggered; When it is determined that the space of the allocated ZG cluster cannot meet the storage demand before the end of the current service cycle, the next reuse cycle is triggered to re-allocate the ZG cluster; When it is determined that there is an unused ZG in the ZG cluster at the end of the current reuse period, the unused ZG is released from the ZG cluster.

8. The method according to claim 1, characterized in that The method further comprises: It is detected that there is a second ZG with an abnormal zone in the ZG cluster of the current reuse period; When it is determined that the second ZG is not in use, reselect a new ZG to replace the second ZG; When it is determined that the second ZG has been used, data is restored for the second ZG when it is determined that a recovery condition is met based on the number of zones in the second ZG where no data loss occurs.

9. The method according to claim 1, characterized in that: In response to the ZG integrity detection instruction, it is detected that there is a second ZG with an abnormal zone in all the allocated ZG clusters; When it is determined that the recovery condition is met according to the number of zones in the second ZG where no data loss occurs, data recovery is performed on the second ZG.

10. A method and device for managing object storage data storage space, characterized in that: The device comprises: A ZG cluster pre-allocation module is used to allocate a ZG cluster matching the data volume according to the estimated data volume of the source data file received in the multiplexing period when the multiplexing period arrives, wherein the ZG cluster includes multiple ZGs, and the number of zones in the ZG is consistent with the number N+M of object shards of the object of the source data file; A data writing module, for responding to the object writing instruction of the source data file, when determining that the current object is the first object in the reuse cycle, writing the current object into one of the ZGs in the ZG cluster; or when determining that the current object is not the first object in the reuse cycle, if the remaining space of the ZG into which the previous object was written is available, writing the current object into the ZG into which the previous object was written, if the remaining space of the ZG into which the previous object was written is not available, reselecting an unused ZG from the ZG cluster for object writing; or if the remaining space of the ZG into which the previous object was written is not available, and there is no unused ZG in the ZG cluster, reselecting N+M unused zones from the storage node to form a ZG for writing object fragments according to the storage node load balancing strategy; or when determining that the current object is the data written for the first time after a previous stream interruption occurs, when determining that the interruption time does not exceed a preset time, obtaining the first ZG into which the last object was written before the interruption occurs, when determining that the remaining space of the first ZG is available, writing the current object into the first ZG, otherwise, reselecting an unused ZG from the ZG cluster for object writing; The ZG cluster deletion module is used to delete any expired ZG or ZG cluster according to the life cycle of the source data file and the writing time of the object segment of the source data file.

11. A method and device for managing object storage data storage space, characterized in that: The device comprises: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor so that the at least one processor can execute the method as described in any one of claims 1-9.

12. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and the computer program is used to enable a computer to execute the method according to any one of claims 1 to 9.

Images