Data processing method and device for object storage

Abstract

The application provides a data processing method for object storage, comprising: providing a plurality of data blocks, a same data block storing a plurality of objects of a same life cycle, each data block corresponding to an index, the index comprising an identifier and a time prefix, the time prefix being a time when a corresponding data block is written; determining an expired index according to the time prefix of each index, to determine an expired data block from the plurality of data blocks; determining a reverse index of metadata of each object in the expired data block according to the identifier in the expired index; deleting the metadata of each object in the expired data block according to the reverse index of the metadata of each object; and deleting the expired data block to release the each object. The technical scheme of the application can reduce the overhead and optimize the disk performance.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer, and in particular, to a data processing method and device for object storage, computer equipment and computer readable storage medium. BACKGROUND

[0002] With the development of computer technology, mass storage, read and write of various data have become the current hot issues. In data storage, object storage as an important data storage method, due to its high reliability, strong scalability, fast access speed and other advantages, has been more and more widely concerned and applied. However, with the continuous growth of data volume, the management and maintenance cost of object storage also increases. In related technologies, object storage generally adopts a life cycle management method, which usually filters objects meeting the recycling conditions, such as expiration time, access time, etc., in all object metadata through a program, then finds the storage location of the data according to the object metadata, performs data deletion and recycling, completes the whole life cycle management of the object, and releases the storage space.

[0003] However, the above management method understood by the present inventors greatly increases hardware overhead and affects read and write performance with the increase of data objects in the storage pool or the dispersion of data objects.

[0004] It should be noted that the above content is not necessarily prior art, and is not used to limit the patent protection scope of the present application. SUMMARY

[0005] An embodiment of the present application provides a data processing method and device for object storage, computer equipment and computer readable storage medium, which are used to solve or alleviate the above problems.

[0006] One aspect of an embodiment of the present application provides a data processing method for object storage, the method comprising:

[0007] providing a plurality of data blocks, a same data block storing a plurality of objects of a same life cycle, each data block corresponding to an index, the index including an identifier and a time prefix, the time prefix being a time when the corresponding data block is written and completed;

[0008] determining an expired index according to the time prefix of each index, to determine an expired data block from the plurality of data blocks;

[0009] determining a reverse index of metadata of each object in the expired data block according to the identifier in the expired index;

[0010] deleting the expired data block to release the each object; and

[0011] According to the reverse index of the metadata of the respective objects, delete the metadata of the respective objects in the expired data block.

[0012] Optionally, the providing the plurality of data blocks comprises:

[0013] Pre-allocating the plurality of data blocks with a fixed size;

[0014] Merging a plurality of objects with the same life cycle and similar upload time into the same data block in a sequential writing manner; wherein the plurality of objects with similar upload time are a plurality of objects uploaded from the client and with a difference in upload time less than a preset difference value.

[0015] Optionally, the plurality of data blocks are distributed in one or more partitions, each partition comprising one or more buckets, the respective buckets in the same partition having the same life cycle, and the bucket being an object container;

[0016] The merging the plurality of objects with the same life cycle and similar upload time into the same data block in a sequential writing manner comprises:

[0017] Dividing the plurality of objects with the same life cycle and similar upload time into a target bucket;

[0018] According to the target bucket, determining a target partition in the plurality of partitions;

[0019] Writing the plurality of objects with the same life cycle and similar upload time into a target data block of the target partition in a sequential writing manner.

[0020] Optionally, the method further comprises:

[0021] In response to the remaining storage capacity of the target data block being less than a preset threshold, creating a target index of the target data block according to a writing time and the life cycle of the target bucket; wherein the writing time is a time when the response is triggered.

[0022] Optionally, the determining the expired index according to the time prefix of each index comprises:

[0023] Determining the expired index based on the time prefix of each index and a pre-set object recycling condition.

[0024] Optionally, further comprising:

[0025] Pre-providing a plurality of offline services, different offline services corresponding to different tasks;

[0026] The tasks comprise determining an expired index, obtaining a reverse index, deleting expired metadata, and deleting an expired data block.

[0027] Optionally, further comprising:

[0028] In the case that the preset backlog state appears in the plurality of offline services, a new offline service is created to increase the number of offline services.

[0029] Another aspect of the embodiments of the present application provides a data processing apparatus for object storage, the apparatus comprising:

[0030] a providing module configured to provide a plurality of data blocks, each data block storing a plurality of objects of a same life cycle, each data block corresponding to an index, the index comprising an identifier and a time prefix, the time prefix being a time when the corresponding data block is written;

[0031] a first determining module configured to determine an expired index according to the time prefix of each index, to determine an expired data block from the plurality of data blocks;

[0032] a second determining module configured to determine a reverse index of metadata of each object in the expired data block according to the identifier in the expired index;

[0033] a first deleting module configured to delete the metadata of each object in the expired data block according to the reverse index of the metadata of each object; and

[0034] a second deleting module configured to delete the expired data block to release the each object.

[0035] Another aspect of the embodiments of the present application provides a computer device, comprising:

[0036] at least one processor; and

[0037] a memory connected with the at least one processor in communication;

[0038] wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0039] Another aspect of the embodiments of the present application provides a computer readable storage medium, the computer readable storage medium storing computer instructions, the computer instructions being executed by a processor to implement the method as described above.

[0040] The data processing method, device, computer device and computer readable storage medium provided by the embodiments of the present application are not dispersedly stored and managed in units of objects, but multiple objects of the same life cycle are combined in one data block. Then, data storage and management are performed in units of data blocks. In the recycling process, expired data blocks are filtered through the time prefix in the index of each data block. The expired data blocks are located through the identifier in the index, and the reverse index of the metadata of each object (expired object) in the expired data blocks is obtained, so that the metadata of the expired object can be reversely deleted, reducing the overhead of filtering the expired object. Since the data blocks are managed in units of data blocks instead of units of objects, the impact of IO on the disk caused by frequent deletion of a large number of objects is alleviated, and the disk fragmentation is alleviated, to a certain extent, the read and write performance of the disk is optimized. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 An operating environment diagram of the data processing method for object storage according to Embodiment One of the present application is schematically shown;

[0042] Figure 2 A flowchart of the data processing method for object storage according to Embodiment One of the present application is schematically shown;

[0043] Figure 3 A sub-flowchart of step S200 is schematically shown;

[0044] Figure 4 A sub-flowchart of step S302 is schematically shown;

[0045] Figure 5 A flowchart of object storage and recycling in an application example is schematically shown;

[0046] Figure 6 A block diagram of the data processing device for object storage according to Embodiment Two of the present application is schematically shown; and

[0047] Figure 7 A hardware architecture schematic diagram of the computer device according to Embodiment Three of the present application is schematically shown. DETAILED DESCRIPTION

[0048] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0049] It should be noted that the terms "first", "second", and the like in the embodiments of the present application are only used for the purpose of description and should not be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined as "first" and "second" can explicitly or implicitly include at least one of the features. In addition, the technical solutions of various embodiments can be combined with each other, but it must be based on the realization of a person skilled in the art, and when the combination of technical solutions contradicts each other or cannot be realized, it should be considered that the combination of technical solutions does not exist and is not within the protection scope required by the present application.

[0050] In the description of the present application, it should be understood that the reference numerals before the steps do not indicate the order of performing the steps before and after, but only for the convenience of describing the present application and distinguishing each step, therefore it should not be understood as a limitation of the present application.

[0051] First, the terms related to the present application are explained:

[0052] Object storage: a computer data storage architecture that manages data as objects, each object usually includes data itself, a certain amount of metadata, and a globally unique identifier.

[0053] Life cycle (TTL, Time To Live): the whole process from creation to recycling and deletion of an object.

[0054] Bucket: a container for storing objects, each bucket has its own storage class, access permission, and region attributes.

[0055] Partition: a property of a bucket, the partition information of a bucket is unique. When uploading an object, it is routed to the corresponding storage node according to the partition of the bucket.

[0056] Index: a storage structure used to quickly find the required data information.

[0057] Secondly, the technical solutions provided by the embodiments of the present application for the convenience of those skilled in the art to understand are described as follows:

[0058] With the development of computer technology, mass storage, read and write of various data have become the current hot issues. In data storage, object storage is an important data storage method. Object storage is a computer data storage architecture, which manages data as objects, each object usually includes data itself, a certain amount of metadata and a globally unique identifier. Life cycle management objects can be used, that is, through a program, objects meeting the recycling conditions are filtered in all object metadata. Then the storage location of the data is found through the metadata of the object to delete and recycle the data, complete the whole life cycle management of the object, and release the storage space.

[0059] However, the present inventors found the following problems: 1. The more the storage pool objects, the more difficult or the greater the overhead of expired object filtering; 2. Object data storage is scattered, and the IO (input / output) generated by frequent deletion of small objects will have a certain impact on disk read and write; 3. Object data storage is scattered, and frequent deletion of small objects will cause disk fragmentation, affecting performance.

[0060] Therefore, the embodiments of the present application provide a data processing scheme for object storage. Objects with the same life cycle and similar upload time are merged into pre-allocated data blocks in a sequential write manner, and the life cycle of the data block is managed to achieve the following technical effects: 1. The reverse index of each object metadata is obtained by traversing the data block, and the object metadata is deleted in reverse, reducing the overhead of filtering expired objects; 2. Recycling in units of data blocks reduces the impact on disk io; 3. Data is merged and written into pre-allocated data blocks, ensuring the sequential nature of data read and write.

[0061] Figure 1 The environment running diagram of the data processing method for object storage according to the embodiments of the present application is schematically shown. As shown in Figure 1 The environment diagram includes a distributed storage system 2, a network 4, and a client 6, wherein:

[0062] The distributed storage system 2, as a storage platform, can be composed of multiple computing devices. The multiple computing devices can include virtualized computing instances. The virtualized computing instances can include virtual machines, such as simulations of computer systems, operating systems, servers, etc. The computing devices can load the virtual machines based on virtual images and / or other data defining specific software (e.g., operating systems, specialized applications, servers) used for the simulation. As the demand for different types of processing services changes, different virtual machines can be loaded and / or terminated on one or more computing devices. A hypervisor can be implemented to manage the use of different virtual machines on the same computing device. The distributed storage system 2 can include different storage functions, such as storing objects, storing metadata, etc.

[0063] The distributed storage system 2 can provide storage, reading, writing, querying, deleting, and the like.

[0064] The distributed storage system 2 can be configured to communicate with the client 6 and the like through the network 4.

[0065] The client 6 can be provided with a user access page for operating the distributed storage system 2 or uploading objects, and the like.

[0066] The following describes a technical implementation scheme provided by the present application through multiple embodiments, taking the distributed storage system 2 as an execution subject.

[0067] Embodiment One

[0068] Figure 2 A flowchart of a data processing method for object storage according to Embodiment One of the present application is schematically shown. As shown in the flowchart, the data processing method for object storage can include steps S200-S208, wherein: Figure 2

[0069] Step S200: A plurality of data blocks are provided, and the same data block stores a plurality of objects of the same life cycle. Each data block corresponds to an index, and the index includes an identifier and a time prefix. The time prefix is the time when the corresponding data block is written and completed.

[0070] Step S202: An expired index is determined according to the time prefix of each index, so as to determine an expired data block from the plurality of data blocks.

[0071] Step S204: According to the identifier in the expired index, a reverse index of the metadata of each object in the expired data block is determined.

[0072] Step S206: According to the reverse index of the metadata of each object, the metadata of each object in the expired data block is deleted.

[0073] Step S208: The expired data block is deleted, so as to release the each object.

[0074] The distributed storage system 2 can be composed of a plurality of storage nodes to form a distributed structure. The distributed storage system can adopt various networking modes, and each storage node is used to provide computing and storage services. The number of storage nodes can be configured according to actual needs. The storage node can be a disk or other non-volatile storage medium. Taking a disk as an example, the storage unit can be a single disk or a disk array formed by a plurality of disks.

[0075] ​The distributed storage system 2 can have multiple storage forms, and the object storage is one of the storage forms. The object storage stores data as objects, and each object can be bound with a unique identifier. This storage form can have high scalability and availability, and supports large data sets. The distributed storage system can be managed by various architectures such as Amazon S3 (an object storage service provided by Amazon).

[0076] For convenience of introduction, some concepts in the distributed storage system 2 are provided as follows:

[0077] Storage object: It is the most basic concept in cloud storage, which refers to the file or data uploaded by the user in the cloud storage. Each object has a unique identifier (Object ID), and the object can be accessed and operated through the identifier.

[0078] Bucket: It is a container in cloud storage, used to store objects. Each bucket has a unique name, and storage objects can be created, deleted and managed in the bucket. The bucket can be regarded as a top-level directory or container, which contains multiple storage objects.

[0079] Partition: It refers to dividing or partitioning a large data set or object set according to a predetermined rule, and each partition can be managed and processed individually. For example, the data can be partitioned according to time, region, user, etc.

[0080] Among them, the partition and the bucket can be logical concepts. The partition can be considered as an attribute of the bucket, and the partition information of the bucket is unique. That is, a bucket is only in one partition, and one partition can include one or more buckets. The partition can logically divide and manage the storage nodes, and one partition can be multiple storage nodes or a part of a storage node. Among them, the data block is a piece of storage segment in the storage node, used to store objects.

[0081] In the embodiment, the multiple data blocks are data blocks in the partition. It should be noted that the multiple data blocks can be located in the same partition or different partitions. Each data block is written in time sequence to multiple objects with the same life cycle.

[0082] Therefore, in the embodiment, the multiple objects with the same life cycle can be combined in one data block, instead of being stored and managed in units of objects. Then, the data is stored and managed in units of data blocks.

[0083] Taking data block A as an example, a plurality of objects with the same life cycle are sequentially written into data block A according to the time sequence of the objects, and an offset is recorded. After data block A is full, no further writing is performed on data block A, and an index (A-index) is generated for data block A. The index (A-index) can be generated based on a plurality of information. For example, the index (A-index) can include time information (time prefix A-time) when data block A is full and an identifier (A-ID) of data block A. The identifier (A-ID) is used to uniquely identify data block A. The time prefix (A-time) is the time when the writing of data block A is completed, and is used to determine the storage time of data block A.

[0084] With the life cycle management of data block A instead of the separate management of the life cycle of each object in data block A, the management efficiency and the disk performance can be improved. The following continues to take data block A as an example for illustrative description.

[0085] The whole process of an object from creation to deletion is referred to as the life cycle of the object. In order to save space, the object whose life cycle ends (expired) needs to be deleted, and the deletion of a large number of expired objects needs to be processed first, and the screening of a large number of expired objects needs a very large overhead. The IO generated by the frequent deletion of a large number of objects has a certain impact on the read and write of the storage medium; the dispersed storage of a large number of objects and the frequent deletion cause the fragmentation of the medium and affect the performance.

[0086] In the above example, a plurality of objects are sequentially merged and written into data block A, so that the deletion (recycling) of the objects can be replaced by the unit of data block instead of the unit of object. The process of deleting each object in data block A is as follows:

[0087] First step: according to the time prefix (A-time) of the index (A-index) of data block A, it is determined whether the index (A-index) is an expired index. If the time prefix (A-time) is before a certain time point, it can be considered as an expired index. When the index (A-index) is an expired index, it is considered that each object in data block A is expired.

[0088] Second step: according to the identifier (A-ID) in the expired index (A-index), the expired data block (data block A) is found.

[0089] Third step: a request for traversing data block A is initiated, and a reverse index of the metadata of all objects in data block A is obtained.

[0090] In object storage, the metadata of an object can include various information of the object, such as name, size, owner, creation time, modification time, location, etc. The reverse index of an object can be found by some attribute values of the metadata.

[0091] For example, if an object is stored in the distributed storage system 2, the system 2 can create an object ID for the object and store the object ID together with other metadata. If the system 2 needs to find the object, the object ID can be used to find the object. In some embodiments, other metadata attributes (e.g., object size or creation time) can also be used to find the object. At this time, the system 2 can use an inverted index to establish a mapping relationship from metadata attribute values to object IDs, so that the object can be found and retrieved more quickly when needed. Therefore, the inverted index of the stored objects is a mechanism that maps metadata attribute values to corresponding object IDs, which makes it more efficient and flexible to find and manage the stored objects in the distributed storage system 2.

[0092] Step 4: deleting expired metadata according to the inverted index of the metadata of the objects.

[0093] After the inverted index of the metadata of all the objects in the data block A is determined, the expired metadata can be deleted. Specifically, the corresponding metadata can be found and deleted according to the information in the inverted index.

[0094] Step 5: deleting the data block A to release all the objects in the data block A.

[0095] It can be seen that whether the data block A is an expired data block A is determined by the index of the data block A. If so, the data block A is found by the identifier in the index. Then the inverted index of the metadata of each object in the data block A is obtained by traversing the data block A, so that the metadata of the expired objects can be deleted in reverse, reducing the overhead of filtering the expired objects; at the same time, the deletion is performed in units of data blocks, reducing the impact on disk IO and alleviating fragmentation. That is, the life cycle management of all the objects in the data block A is performed in units of data blocks A, improving the efficiency.

[0096] In optional embodiments, as shown in FIG. 2, the step S200 of “providing a plurality of data blocks” can include: Figure 3

[0097] Step S300, pre-allocating a plurality of data blocks of a fixed size;

[0098] Step S302, merging a plurality of objects with the same life cycle and similar upload time into the same data block in a sequential writing manner; wherein the plurality of objects with similar upload time are a plurality of objects uploaded from a client and having a difference in upload time less than a preset difference.

[0099] ​The multiple objects with the same life cycle and similar upload time are sequentially written into the same data block. If the data block storage space is full, the subsequent objects will be written into the next data block according to the same logic. In time sequence, the multiple objects are written into the pre-allocated data block in a combined manner, which ensures the sequential nature of data reading and writing.

[0100] In some other embodiments, since the storage time corresponding to different life cycles is different, for the objects with the same life cycle, different data block sizes can be set according to the life cycle length to improve the storage efficiency.

[0101] In optional embodiments, the multiple data blocks are distributed in one or more partitions, and each partition includes one or more buckets. The life cycles of the buckets in the same partition are the same, and the bucket is an object container. Figure 4 As shown, the step S302 of "combining the multiple objects with the same life cycle and similar upload time into the same data block in a sequential writing manner" can include:

[0102] The step S400 divides the multiple objects with the same life cycle and similar upload time into the target bucket.

[0103] The step S402 determines the target partition in the multiple partitions according to the target bucket.

[0104] The step S404 writes the multiple objects with the same life cycle and similar upload time into the target data block of the target partition in a sequential writing manner.

[0105] As an example, the multiple objects with the same life cycle and similar upload time are divided into the target bucket. The bucket is an object container, and each bucket has its own storage class, access permission, and partition attribute. Therefore, according to the partition attribute information of the target bucket, the target partition can be determined, and the multiple objects are routed to the target data block of the target partition. It can be known that the above scheme can effectively combine the multiple objects with the same life cycle and similar upload time to write together, so that the object life cycle management based on the data block unit can be realized.

[0106] The target partition can correspond to multiple data blocks. For the target partition, the multiple objects can be sequentially written into the target data block. When the target data block is full or meets certain conditions (such as the writing time interval exceeds the time threshold), the target data block is switched to the next available data block. In the next data block, the remaining objects with the same life cycle and similar upload time are written into the next data block in a sequential writing manner.

[0107] The plurality of partitions can be adjusted according to configuration parameters. One partition can correspond to a plurality of storage nodes or a part of one storage node. Different storage nodes can correspond to different storage capacities and bandwidths. To effectively utilize resources, a dynamic partition strategy can be adopted to adjust the partition size and the number of partitions according to the load condition. For example, a load balancing algorithm can be adopted to distribute storage requests to different nodes according to the load condition of the nodes.

[0108] In an optional embodiment, the method for processing data stored by objects can further include:

[0109] In response to the remaining storage capacity of the target data block being less than a preset threshold, a target index of the target data block is created according to a write time and a life cycle of the target data block; wherein the write time is the time when the response is triggered.

[0110] In an exemplary application, the storage capacity of the target data block is monitored, and if the remaining storage capacity is less than a preset threshold, a target index of the target data block is created as a condition according to the life cycle of the target bucket and the write time. The target index can include various attribute information of the target data block, such as a unique identifier, a write time (time prefix), an object size, an object offset, a life cycle, etc. The unique identifier can be a hash value, etc.

[0111] In an optional embodiment, the step S202 of "determining expired indexes according to the time prefix of each index" can include determining the expired indexes based on the time prefix of each index and a preset object recycling condition.

[0112] In an exemplary application, the object recycling condition can include a time interval corresponding to the life cycle. For example, when a bucket is created 15 days ago and the life cycle of the bucket is 15 days, if the object recycling condition is 15 days, the indexes of the data blocks used to store the objects of the bucket can be pulled and identified as expired indexes. It can be seen that the expired indexes can be quickly and effectively pulled through the preset object recycling condition.

[0113] In an optional embodiment, the method for processing data stored by objects can further include:

[0114] A plurality of offline services are provided in advance, and different offline services correspond to different tasks;

[0115] The tasks include determining expired indexes, obtaining reverse indexes, deleting expired metadata, and deleting expired data blocks.

[0116] In an exemplary application, a plurality of offline services are pulled in advance, and each offline service corresponds to a task.

[0117] For the task of determining expired indexes, the offline service is used to traverse all indexes to determine expired indexes according to time prefixes and object recycling conditions.

[0118] For the task of obtaining reverse indexes, the offline service is used to determine expired data blocks according to identifiers in the expired indexes, and then traverse the expired data blocks to obtain reverse indexes of metadata of each object in the expired data blocks.

[0119] For the task of deleting expired metadata, the offline service is used to find expired metadata through the obtained reverse indexes and delete the expired metadata in batches.

[0120] For the task of deleting expired data blocks, the offline service is used to delete the expired data blocks to release the objects in the expired data blocks.

[0121] The above services are only exemplary. Other services can also be set according to actual conditions. In some embodiments, the number and resource allocation of services can be dynamically adjusted according to the type and complexity of tasks to improve processing efficiency. Task priority and scheduling mechanisms can also be added to the offline service to meet the priority and timeliness requirements of different tasks.

[0122] In optional embodiments, the data processing method for object storage can further include:

[0123] In the case where the plurality of offline services appear a preset backlog state, a new offline service is created to increase the number of offline services.

[0124] In an exemplary application, the status of each offline service, such as working status, task completion degree, remaining storage space, etc., can be monitored; when a plurality of offline services appear a preset backlog state, a new offline service is created; after the new offline service is created, a task is allocated to it; part of the unfinished tasks are moved from the old offline service to the new offline service; the new offline service is added to the offline service pool to continue to execute tasks and improve service efficiency. It should be noted that the preset backlog state can refer to a preset number of tasks appearing queuing and processing delay.

[0125] In order to make the embodiments of the present application easier to understand, the following will be combined with Figure 5 An exemplary application is provided.

[0126] First, object storage strategy.

[0127] A plurality of partitions and a plurality of buckets are set in advance by configuring parameters. One partition corresponds to a plurality of buckets, each partition corresponds to one or more storage nodes, and each storage node has a plurality of data blocks of a fixed size.

[0128] The client routes objects to corresponding storage nodes through the partition of the bucket.

[0129] The bucket is a container of objects, and the partition is a property of the bucket.

[0130] After the bucket is created, it is known that the objects in the bucket are stored in which storage node.

[0131] After the storage node is determined, the multiple objects are aggregated into the pre-allocated data block in the storage node, and the offset is recorded.

[0132] It should be noted that the life cycle of all objects in the same bucket is the same.

[0133] For example, the life cycle of object 1 is 15 days, the life cycle of object 2 is 2 days, the life cycle of object 3 is 15 days, the life cycle of object 4 is 15 days, and the life cycle of object 5 is 15 days. Object 1, 3, 4, and 5 can be placed in the same bucket. It can be seen that the life cycle of this bucket is actually 15 days.

[0134] In addition, the upload time of objects 1, 3, 4, and 5 should also be the same or close.

[0135] In this way, when objects 1, 3, 4, and 5 are stored in the same data block, the data block can be deleted after 15 days to release these objects.

[0136] In addition, the life cycle of the bucket of all objects in the same partition is also the same, and the partition is a time partition.

[0137] Second: Object deletion (recycle) trigger logic.

[0138] S500: The client uploads the object to the storage node under the partition according to the partition where the bucket is located, and sequentially writes the pre-allocated fixed-size data block. The metadata of each object is stored in the metadata storage node.

[0139] S502: When the data block is full, create an index based on the current time and the life cycle of the bucket; wherein the index includes a time prefix and the unique ID information of the data block, and the time prefix corresponds to the time when the data block is full.

[0140] S504: The offline service pulls the expired index that meets the recycling condition according to the time prefix.

[0141] If there is a backlog of index service, increase the number of offline services.

[0142] The recycling condition in the related art includes: scanning all objects in the storage cluster, judging whether to delete according to the creation time and the current time; or establishing an index (including a timestamp, a key of the object and the like) for each object, and the index with the timestamp is ordered. Compared with the recycling operation in the related art, in the technical solution in the application example, the scanning can be performed according to the prefix matching, without scanning all objects.

[0143] S506: Find the corresponding storage node through the data block unique ID recorded in the expired index, and initiate a request for traversing the expired data block corresponding to the expired index, to obtain the reverse index of the metadata of all objects in the expired data block.

[0144] S508: According to the reverse index, request the metadata storage service to batch delete the metadata of all objects in the expired data block.

[0145] S510: Delete the expired data block to release the storage space, thereby completing the lifecycle management of all objects in the batch.

[0146] In the technical solution in the application example, the bucket can be understood as a directory for managing metadata, in the recycling process, the time prefix of the metadata is scanned, and then the corresponding physical storage location is found according to the metadata information recorded in the metadata for deletion. There is a lag from the expiration of the lifecycle to the real release of the storage space, and after the expiration, only the metadata needs to be marked to make the user perceive that the object has been deleted, and actually there are still some processes behind.

[0147] In the technical solution in the application example, objects with the same lifecycle and similar upload time are aggregated into a fixed-size data block, and after the data block is full, the ID of the data block, the timestamp when the data block is full (that is, the deletion of the object stored before the data block is full is lagged), and the lifecycle of the batch of objects are created. This deletion in the unit of data block can delete a batch of objects at a time, so that the storage space release efficiency is higher.

[0148] Embodiment Two

[0149] Figure 6 A block diagram of a data processing apparatus for object storage according to Embodiment Two of the present application is schematically shown, which can be divided into one or more program modules, the one or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiments of the present application. The program modules referred to in the embodiments of the present application refer to a series of computer program instruction segments capable of completing a specific function, and the functions of the program modules of the embodiments will be specifically described below. For example, Figure 6As shown, the data processing apparatus 600 for object storage can include a providing module 610, a first determining module 620, a second determining module 630, a first deleting module 640, a second deleting module 650, wherein:

[0150] The providing module 610 is configured to provide a plurality of data blocks, wherein the same data block stores a plurality of objects with the same life cycle, each data block corresponds to an index, and the index includes an identifier and a time prefix, wherein the time prefix is the time when the corresponding data block is written and completed.

[0151] The first determining module 620 is configured to determine an expired index according to the time prefix of each index, so as to determine an expired data block from the plurality of data blocks.

[0152] The second determining module 630 is configured to determine a reverse index of the metadata of each object in the expired data block according to the identifier in the expired index.

[0153] The first deleting module 640 is configured to delete the metadata of each object in the expired data block according to the reverse index of the metadata of each object.

[0154] The second deleting module 650 is configured to delete the expired data block to release the objects.

[0155] In an optional embodiment, the providing module 610 is further configured to:

[0156] Pre-allocate the plurality of data blocks with a fixed size.

[0157] Merge a plurality of objects with the same life cycle and similar upload time into the same data block in a sequential writing manner, wherein the plurality of objects with similar upload time are a plurality of objects uploaded from the client and with a difference in upload time less than a preset difference.

[0158] In an optional embodiment, the plurality of data blocks are distributed in one or more partitions, each partition includes one or more buckets, the life cycles of the buckets in the same partition are the same, and the bucket is an object container.

[0159] The providing module 610 is further configured to:

[0160] Divide a plurality of objects with the same life cycle and similar upload time into a target bucket.

[0161] Determine a target partition in the plurality of partitions according to the target bucket.

[0162] Write a plurality of objects with the same life cycle and similar upload time into a target data block of the target partition in a sequential writing manner.

[0163] In an optional embodiment, the apparatus further comprises a creating module (not identified) configured to:

[0164] In response to the remaining storage capacity of the target data block being less than a preset threshold, a target index of the target data block is created according to a write time and a life cycle of the target data block, wherein the write time is the time when the response is triggered.

[0165] In an optional embodiment, the first determining module 620 is configured to:

[0166] The expired index is determined based on a time prefix of each index and a preset object recycling condition.

[0167] In an optional embodiment, the apparatus further comprises a service providing module (not identified) configured to:

[0168] A plurality of offline services are provided in advance, and different offline services correspond to different tasks.

[0169] The tasks include determining an expired index, obtaining a reverse index, deleting an expired metadata, and deleting an expired data block.

[0170] In an optional embodiment, the apparatus further comprises a service creating module (not identified) configured to:

[0171] In a case where the plurality of offline services are in a preset backlog state, a new offline service is created to increase the number of offline services.

[0172] Embodiment Three

[0173] Figure 7 A hardware architecture schematic diagram of a computer device 10000 suitable for implementing the data processing method for object storage according to Embodiment Three of the present application is schematically shown. The computer device 10000 can be part of the distributed storage system 2. In this embodiment, the computer device 10000 can be a rack-mounted server, a blade server, a tower server, or a cabinet server (including a standalone server, or a server cluster composed of multiple servers), etc. As shown in the figure, the computer device 10000 at least includes but is not limited to a memory 10010, a processor 10020, and a network interface 10030 which can be communicatively linked through a system bus. Among them: Figure 7

[0174] ​The memory 10010 includes at least one type of computer-readable storage media, such as a flash memory, a hard disk, a multimedia card (e.g., SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the memory 10010 can be an internal memory module of the computer device 10000, such as a hard disk or a memory of the computer device 10000. In other embodiments, the memory 10010 can also be an external memory device of the computer device 10000, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like. Of course, the memory 10010 can include both an internal memory module and an external memory device of the computer device 10000. In this embodiment, the memory 10010 is generally used to store an operating system and various application programs installed in the computer device 10000, such as program codes of the data processing method for objects, and the like. In addition, the memory 10010 can also be used to temporarily store various data that have been output or will be output.

[0175] The processor 10020 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other object processing chips in some embodiments. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication of the computer device 10000, and the like. In this embodiment, the processor 10020 is used to run program codes or process data stored in the memory 10010.

[0176] The network interface 10030 can include a wireless network interface or a wired network interface, and is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 with an external terminal through a network, to establish a data transmission channel and a communication link between the computer device 10000 and the external terminal, and the like. The network can be an Intranet, the Internet, a Global System of Mobile communication (GSM), a Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, Bluetooth, Wi-Fi, or other wireless or wired networks.

[0177] It should be noted that, Figure 7 Only the computer device with components 10010-10030 is shown, but it should be understood that all the shown components are not required to be implemented, and more or fewer components can be alternatively implemented.

[0178] In this embodiment, the data processing method for object storage stored in the memory 10010 can also be divided into one or more program modules, and executed by one or more processors (in this embodiment, the processor 10020) to complete the data processing method for object storage in the embodiments of the present application.

[0179] Embodiment Four

[0180] The embodiments of the present application also provide a computer readable storage medium, and the computer readable storage medium has a computer program stored thereon. The computer program is executed by a processor to implement the steps of the data processing method for object storage in the embodiments.

[0181] In this embodiment, the computer readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the computer readable storage medium can be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device. In other embodiments, the computer readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device. Of course, the computer readable storage medium can also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer readable storage medium is usually used to store an operating system and various application software installed on the computer device, such as program codes of the data processing method for object storage in the embodiments, etc. In addition, the computer readable storage medium can also be used to temporarily store various data that have been output or will be output.

[0182] Obviously, those skilled in the art should understand that each module or each step of the above-mentioned embodiments of the present application can be realized by a general computing device, which can be concentrated on a single computing device or distributed on a network composed of multiple computing devices, and optionally, each module or each step can be realized by program codes executable by a computing device, so that each module or each step can be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described can be executed in different order, or each module or each step can be manufactured into an individual integrated circuit module, or multiple modules or steps can be manufactured into a single integrated circuit module. Therefore, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0183] It should be noted that the above is only the preferred embodiment of the present application, and does not limit the patent protection scope of the present application, and any equivalent structure or equivalent process transformation using the content of the specification and drawings of the present application, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A data processing method for object storage, characterized in that, The method comprises: providing a plurality of data blocks, a same data block storing a plurality of objects of a same life cycle, each data block corresponding to an index, the index comprising an identifier and a time prefix, the time prefix being a time when writing of the corresponding data block is completed; determining an expired index according to the time prefix of each index, to determine an expired data block from the plurality of data blocks; determining a reverse index of metadata of each object in the expired data block according to the identifier in the expired index; deleting the metadata of each object in the expired data block according to the reverse index of the metadata of each object; and deleting the expired data block to release the objects.

2. The method of claim 1, wherein, The providing of the plurality of data blocks comprises: pre-allocating the plurality of data blocks of a fixed size; merging a plurality of objects of a same life cycle and close upload time into a same data block in a sequential writing manner; wherein the plurality of objects of close upload time are a plurality of objects uploaded from a client and having a difference in upload time less than a preset difference value.

3. The method of claim 2, wherein, The plurality of data blocks are distributed in one or more partitions, each partition comprising one or more buckets, the life cycles of the buckets in a same partition being the same, and a bucket being an object container; the merging of the plurality of objects of a same life cycle and close upload time into a same data block in a sequential writing manner comprises: dividing the plurality of objects of a same life cycle and close upload time into a target bucket; determining a target partition from the plurality of partitions according to the target bucket; writing the plurality of objects of a same life cycle and close upload time into a target data block of the target partition in a sequential writing manner.

4. The method of claim 3, wherein, The method further comprises: in response to a remaining storage capacity of the target data block being less than a preset threshold, creating a target index of the target data block according to a writing time and a life cycle of the target bucket; wherein the writing time is a time when the response is triggered.

5. The method according to any one of claims 1 to 4, characterized in that, The determining of the expired index according to the time prefix of each index comprises: determining the expired index based on the time prefix of each index and a preset object recycling condition.

6. The method according to any one of claims 1 to 4, characterized in that, Further comprising: providing a plurality of offline services in advance, different offline services corresponding to different tasks; wherein the tasks comprise determining an expired index, obtaining a reverse index, deleting expired metadata, and deleting an expired data block.

7. The method of claim 6, wherein, Further comprising: in a case where the plurality of offline services are in a preset backlog state, creating a new offline service to increase the number of offline services.

8. A data processing apparatus for an object store, the data processing apparatus comprising: The apparatus comprises: a providing module configured to provide a plurality of data blocks, a same data block storing a plurality of objects of a same life cycle, each data block corresponding to an index, the index comprising an identifier and a time prefix, the time prefix being a time when writing of the corresponding data block is completed; a first determining module configured to determine an expired index according to the time prefix of each index, to determine an expired data block from the plurality of data blocks; a second determining module configured to determine a reverse index of metadata of each object in the expired data block according to the identifier in the expired index; a first deleting module configured to delete the metadata of each object in the expired data block according to the reverse index of the metadata of each object; and a second deleting module configured to delete the expired data block to release the objects. A second deleting module is configured to delete the expired data block to release the respective object.

9. A computer device, comprising: The application relates to a computer readable storage medium storing computer instructions, wherein the computer instructions are executed by a processor to implement the method according to any one of claims 1-7. The application relates to a computer readable storage medium storing computer instructions, wherein the computer instructions are executed by a processor to implement the method according to any one of claims 1-7. The application relates to a computer readable storage medium storing computer instructions, wherein the computer instructions are executed by a processor to implement the method according to any one of claims 1-7. The application relates to a computer readable storage medium storing computer instructions, wherein the computer instructions are executed by a processor to implement the method according to any one of claims 1-7. ​ 10. A computer-readable storage medium, characterized in that, ​ 11. A computer program product comprising computer instructions, characterized in that, ​

Images