Space arrangement method for distributed object storage

Abstract

The present application belongs to the field of data storage, and relates to a space arrangement method for distributed object storage. The method comprises: acquiring, as source objects, aggregation large objects meeting a hole ratio requirement, and inserting, into a hash list, source objects, fragment spaces of which meet a hole minimum threshold; selecting, from the hash list, the source objects having the highest fragment rate to undergo space arrangement, creating a target aggregation large object on the basis of requirements and using the target aggregation large object as a target object; generating a target object mapping relationship table; migrating data of small objects on the source objects to the target object on the basis of the target object mapping relationship table; after migration is successful, updating layout information corresponding to metadata of the small objects themselves; and executing a cleaning operation, so as to end the task. In the method of the present application, by means of migrating undeleted small objects from original large objects onto a new empty large object, the fragmentation and waste of a storage space are effectively reduced, thereby better improving the utilization rate of the storage space.

Description

A distributed object storage space organization method

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on December 13, 2023, with application number 202311709521X, entitled “A Method for Organizing Distributed Object Storage Space,” the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the field of data storage, and in particular to a method for organizing distributed object storage space. Background Art

[0003] (1) Existing distributed object storage has different definitions for the sizes of large objects and small objects, and also handles uploading large objects and small objects differently.

[0004] For small object uploads, most support aggregating small objects on SSD disks into a large object and then storing it on the backend HDD disk. This not only improves disk space utilization, but also enhances the performance of small object uploads. However, after deleting small objects, many discontinuous empty spaces will be left in the large object space generated by the aggregation, forming fragmented data blocks. This increases latency overhead when randomly reading non-deleted objects, resulting in a poor user experience for upper-layer applications. Deleting and releasing the storage space occupied by aggregated small objects may cause the storage space to become discontinuous, that is, a large amount of non-adjacent free space appears. This may lead to fragmentation of the storage system, reduce data reading and writing efficiency, and affect system availability. To solve this problem, a common optimization method is to read the empty space and valid data on each aggregated large object, and then migrate all valid data to form continuous data blocks for reading.

[0005] (2) Existing fragmented space optimization methods, while addressing the issue of valid data continuity on aggregated large objects, incur significant memory overhead when processing all aggregated objects, and still waste storage space at the end of the aggregated objects. This can lead to resource competition with other processes when performing space cleanup. Scattered holes can increase the complexity of metadata management, as the system needs to maintain metadata information for non-hole objects to ensure correct object location and management.

[0006] (3) For video surveillance data storage, when small objects smaller than 1MB, such as logs, text, and images, are deleted, holes of varying sizes are left in the aggregated large object storage space. These holes can lead to a waste of storage space because the system cannot effectively reuse the space, thereby increasing storage costs. Summary of the Invention

[0007] In view of the above shortcomings of the existing technology, the purpose of the invention is to provide a distributed object storage space organization method, which effectively reduces the fragmentation and waste of storage space by migrating the undeleted small objects from the original large object to the new empty large object, and better improves the storage space utilization.

[0008] This application proposes a distributed object storage space organization method, including:

[0009] S1: Obtain aggregated large objects that meet the hole rate requirement as source objects, and insert the source objects whose fragment space meets the minimum hole threshold into the hash list;

[0010] S2, selecting the source object with the highest fragmentation rate from the hash list for space arrangement, and creating a target aggregate large object as the target object according to demand;

[0011] S3, traversing the object table of the source object in batches, obtaining offset information of small objects on the source object, and generating an ordered object table after sorting. According to the ordered object table, obtaining interval information of each small object on the large object, adding the interval information to the interval mapping table of the source object, assigning corresponding target object intervals to the source object intervals, and then generating a target object mapping relationship table;

[0012] S4, migrating the small object data on the source object to the target object according to the target object mapping relationship table, wherein:

[0013] First, data migration is performed, including: reading the interval data corresponding to the small objects in the source object, and then writing it to the new interval marked by the target object. After receiving the data write success response, it indicates that the data migration is successful;

[0014] Then, metadata and object table migration is performed, including: inserting the small object list into the object table LSM tree corresponding to the target object in batches, and inserting the metadata of the source object after interval sorting into the metadata LSM tree of the target object;

[0015] S5: After the migration is successful, the small objects in the source object are read in batches, and the interval layout information corresponding to the metadata of the small objects themselves is updated according to the interval layout information corresponding to the small objects on the target object;

[0016] S6, perform cleanup operations and end the task.

[0017] Furthermore, the minimum hole threshold is dynamically adjusted according to the cluster capacity utilization rate.

[0018] Furthermore, each source object can only select one target object, and each target object corresponds to multiple source objects.

[0019] Furthermore, in S2, space consolidation begins, the target object is automatically created in the bucket, and the states of the source object and the target object are set to space consolidation states, wherein the sum of the effective capacities of the source objects cannot exceed the capacity of the target object.

[0020] Furthermore, in S3, the object table information on the source object is stored in the LSM tree, and the object table and related interval memory on the source object are obtained through the LSM enumeration interface.

[0021] Furthermore, in S3, the offset information of the small objects on the source object is obtained, and the offset information of the small objects on the source object is obtained according to the object name and segment number, and the small objects are sorted from large to small to generate an ordered object table.

[0022] Furthermore, in S3, after reading the interval mapping table of the source object, the offset information and interval layout of the small object are modified to form a continuous address, and the target object mapping relationship table is generated.

[0023] Furthermore, in S5, before updating the layout information of the small information object, it is first checked whether the small object still exists. If it does not exist, it means that it has been deleted during the space arrangement, and there is no need to update the layout information.

[0024] Furthermore, in S5, when updating the layout information of the small information object, the update is performed in the order of the offset information of the small object, and it is determined whether the update process is normal. If it is normal, the memory is solidified after the update to ensure that the subsequent sorting task can be continued from the failed offset when an exception occurs; if it is an exception, it means that the layout information of some objects is updated successfully, and the source object and the target object both have valid data, which are used as the source object for the next spatial sorting.

[0025] Furthermore, in step S6, the source objects are cleaned up, the solidified memory structure of the space arrangement is cleaned up, and the solidified space arrangement tasks are cleaned up.

[0026] The beneficial effects of this application are as follows:

[0027] (1) Dynamic migration strategy for small objects: By continuously integrating valid data segments in large objects with a large gap between valid space and allocated space into new empty large objects, the purpose of eliminating holes is achieved, while also reducing the impact on system load when performing space sorting. This dynamic adjustment and migration strategy can dynamically adjust the threshold according to cluster capacity usage and system load, and automatically start background space sorting tasks. This optimization strategy allows the system to maintain high efficiency under different loads, and solves the problem of increased storage space fragmentation caused by repeated writing and deletion, or overwriting of massive small objects, resulting in high complexity of metadata management and affecting object reading and writing efficiency.

[0028] (2) Object metadata maintenance method: The metadata and object table maintenance method ensures the consistency and integrity of data and metadata before and after migration by establishing an interval mapping table. By establishing an interval mapping table, data migration is performed while also migrating metadata and object tables, ensuring data consistency and integrity.

[0029] (3) Space consolidation control structure: Adaptive dynamic adjustments are made based on cluster capacity utilization and load, including task concurrency, minimum space threshold, and task execution interval, to improve space consolidation efficiency and data storage continuity. By migrating undeleted small objects from the original large object to a new empty large object, storage space fragmentation and waste are effectively reduced, thereby improving storage space utilization.

[0030] (4) Storage Space Optimization: Existing technologies eliminate storage holes by migrating data within the original large object's own space. This application effectively reduces storage space fragmentation and waste by migrating undeleted small objects from the original large object to a new, empty large object, thereby improving storage space utilization. This application can help users and enterprises use storage space more efficiently, reduce fragmentation, and lower storage costs.

[0031] (5) Adaptive space arrangement control structure: Adaptively adjust various parameters of background tasks, including concurrency, task interval, etc., according to cluster capacity usage and system load.

[0032] (6) Data continuity: Optimizing storage space helps create continuous data storage blocks, thereby reducing random access overhead and speeding up read and write operations.

[0033] (7) Simplified operation and maintenance: The system supports turning on and off background space cleaning tasks at any time. Once turned on, various parameters are adaptively adjusted according to the current situation of the cluster to improve the efficiency of void elimination. No complex operations are required to monitor and manage tasks. Simple query commands are required to view the intermediate process and results. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings are only for the purpose of illustrating specific embodiments and are not to be considered as limiting the present application. Throughout the drawings, the same reference numerals represent the same components. Obviously, the drawings described below are only some of the embodiments described in the present application. Those skilled in the art can also derive other drawings based on these drawings.

[0035] FIG1 is a flow chart of a method for organizing a distributed object storage space according to an embodiment of the present application;

[0036] FIG2 is a schematic diagram of a distributed object storage space organization method according to an embodiment of the present application;

[0037] FIG3 is a schematic diagram of a source object space organization result according to an embodiment of the present application;

[0038] FIG4 is a flowchart of the interval sorting workflow according to an embodiment of the present application;

[0039] FIG5 is a flowchart of an object migration workflow according to an embodiment of the present application;

[0040] FIG6 is a diagram of a layout update workflow according to an embodiment of the present application;

[0041] FIG7 is a flowchart of a space clearing process according to an embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without making creative work should fall within the scope of protection of this application.

[0043] Furthermore, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts disclosed in this application.

[0044] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on this application. In addition, the terms "first", "second" and "third" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance. The terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be a communication between the internal parts of two elements. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0045] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of methods and systems consistent with certain aspects of the present application, as detailed in the appended claims.

[0046] The following describes the technical terms involved in the distributed object storage space organization method according to the embodiment of the present application:

[0047] GC: Garbage Collection (GC), in distributed object storage, GC generally refers to some asynchronous disk space recovery or space arrangement operations. When the client executes the operation of deleting objects or overwriting objects, the disk space occupied by the corresponding objects will not be released immediately, but will be handed over to the background GC module for processing.

[0048] LSM tree: Log-Structured Merge Tree. It is a data structure used to implement key-value storage systems and is widely used in distributed databases and distributed file systems. Its core concept is to sequentially append key-value pairs to multiple storage files at different levels on disk.

[0049] As shown in FIG1 and FIG2 , the distributed object storage space organization method according to an embodiment of the present application includes:

[0050] S1, obtains the aggregated large object that meets the hole rate requirement as the source object, and inserts the source object whose fragment space meets the minimum hole threshold into the hash list.

[0051] For a distributed object storage cluster, clients upload objects of different sizes to the cluster. Objects smaller than 1M are called small objects. After being uploaded, they are aggregated into large objects in the cache space composed of SSD disks and stored on the backend HDD disks. The maximum size of the aggregated large object is 96M or the number of aggregated objects does not exceed 36,000. As the cluster runs for longer, the number of uploaded small objects and generated aggregated large objects also increases. When the client executes an object deletion operation, some discontinuous empty spaces will be generated on the corresponding large object, and the deleted object space will become fragmented space. The space compaction thread continues to scan in the background. When the empty space on the aggregated large object reaches a certain threshold compared to the effective capacity, the space compaction execution is automatically triggered based on the usage capacity ratio of the cluster storage pool.

[0052] To avoid the need to read the source object's metadata information from the disk every time when selecting a source object for space compaction, the metadata management module inserts large objects that are in the aggregated state and whose fragmentation space meets the minimum void threshold into a hash table. When selecting source objects, it is only necessary to traverse the hash table and select the objects with the highest void rate for compaction.

[0053] In the embodiment of the present application, the minimum threshold for holes is dynamically adjusted based on the cluster capacity utilization rate. The higher the utilization rate, the lower the minimum threshold, ensuring that when the cluster space is available, the front-end business is minimally affected by the back-end space arrangement.

[0054] S2 selects the source object with the highest fragmentation rate from the hash list for space compaction, and creates a target aggregate large object as the target object based on demand.

[0055] Specifically, space cleanup begins, and the target object is automatically created in the bucket. The source and target objects are set to the space cleanup state. A background thread ensures that the source object does not accept other migration operations, and the target object does not accept new object aggregation writes. During the space cleanup, the source object can be read and deleted, but not written.

[0056] As shown in Figure 3, each source object can only select one target object, and each target object corresponds to multiple source objects. That is, multiple source objects can select the same target object. The sum of the effective capacity of multiple source objects cannot exceed the capacity of the target object.

[0057] S3, traverse the object table of the source object in batches, obtain the offset information of the small objects on the source object, and generate an ordered object table after sorting. According to the ordered object table, obtain the interval (layout) information of each small object on the large object, add the interval information to the interval mapping table of the source object, assign the corresponding target object interval to the source object interval, and then generate the target object mapping relationship table.

[0058] As shown in Figure 4, the object table information on the source object is first stored in the LSM tree. The object table and the memory of the related intervals on the source object are obtained through the LSM enumeration interface. Specifically, the metadata of the source object and the corresponding object table are read, and the layout information of the object metadata recorded in the object table is read in batches and stored in a defined interval mapping table. Because the number of small objects in the object table may be large, to avoid insufficient memory, the object table is traversed in batches, enumerating a maximum of N objects at a time. Then, based on the object name and segment number, the offset information of the small objects on the source object is obtained, and the small objects are sorted from largest to smallest to generate an ordered object table.

[0059] Then, based on the object information in the ordered object table, the interval information of each small object on the large object is obtained, that is, the interval (layout) information of each small object on the large object is obtained.

[0060] If the acquisition is successful, the interval information is added to the interval mapping table of the source object, and then the corresponding target large object interval is allocated according to the interval, and the corresponding target object interval is allocated to the source object interval; otherwise, it is determined whether the object is deleted and whether there is a segment number record.

[0061] This step traverses the selected small objects on the source object in batches to avoid insufficient memory caused by traversal, and then adds the interval information to the interval mapping table of the source object.

[0062] Finally, the interval mapping table of the source object is read, the offset information and layout of the object are modified to form a continuous address, and a new target object mapping relationship table is generated.

[0063] Specifically, the offset information of the small object is modified, that is, the offset information of the small object on the target large object is modified, and then the layout information or length information of the small object is read. In this way, multiple small objects are stored in the target large object in an orderly manner.

[0064] S4: Migrate the small object data on the source object to the target object according to the target object mapping relationship table.

[0065] As shown in Figure 5, after interval organization, the source intervals of the source object to be migrated and the target interval mapping tables of the target object are saved. Based on the generated target object mapping table, the mapping migration from the source object to the corresponding interval of the target object is performed. Based on the source object's object interval table, each object's data is migrated to the contiguous address space defined in the target object table.

[0066] Target interval mapping table: refers to the intervals (layout) corresponding to the small objects to be migrated on the target large object, which is an ordered list formed by the intervals of multiple small objects.

[0067] Object interval table: refers to the interval where small objects on the source object (aggregate large object) are located on the source object, including offset information and length (layout) information.

[0068] Target object table: refers to multiple target objects recorded in the hash list, which are also managed by a key-value table.

[0069] The relationship between the above three types of tables is as follows: the object interval table is the interval corresponding to the small object on the large object, the target object table records the metadata related information of the target large object, and the target interval mapping table refers to the interval mapping relationship between the small object on the source object and the target large object.

[0070] If a migration fails, a retry will be performed. If the retry still fails, the decision on whether to destroy the target object or just update its state will be made based on whether valid data already exists on the target object. Subsequent resource cleanup should also be performed promptly.

[0071] The migration in this step includes: data migration, metadata and object table migration.

[0072] (1) Data migration

[0073] First, perform data migration, convert the source / target intervals of the object to be migrated into the intervals corresponding to the underlying disk data distribution, and start the data migration process

[0074] Specifically, the data migration task is executed by the background migration thread, which reads the interval data corresponding to the small objects in the source object and then writes it to the new interval marked by the target object. After receiving the data write success response after the data is written to the disk, it indicates that the data migration is successful.

[0075] (2) Metadata and object table migration

[0076] Then, after the data migration is completed, the metadata and object table migration are performed, the metadata of the target large object is updated, and the small object list is inserted into the object table LSM tree corresponding to the target object in batches.

[0077] Specifically, after the small object data corresponding to the source object is successfully migrated, the small object list is batch inserted into the object table LSM tree corresponding to the target object. At the same time, the metadata of the source object after interval sorting is also inserted into the metadata LSM tree of the target object.

[0078] S5: After the migration is successful, the small objects in the source object are read in batches. Based on the layout information corresponding to the small objects on the target object, the layout information corresponding to the metadata of the small objects themselves is updated.

[0079] As shown in Figure 6, after data and metadata migration is complete, the layout information of the small objects needs to be updated synchronously. This means reading the small objects in the source object and updating the layout information of the small objects' metadata based on the layout information of the small objects on the target object to ensure metadata consistency.

[0080] Before updating the layout information of a small information object, read the object table of the source object to check whether the small object still exists. If it does not exist, it means that it was deleted during space organization, and there is no need to update the layout information.

[0081] The target object interval table is read and objects are acquired in batches for update. When updating the layout information of small objects, the updates are performed in order of their offset information. The normality of the update process is determined. If normal, the memory is solidified after the update to ensure that subsequent cleanup tasks can resume from the failed offset in the event of an exception. If an exception occurs, it indicates that the layout information of some objects was successfully updated. Both the source and target objects, which contain valid data, will be used as the source objects for the next cleanup. This is because the currently created target large object will become the source object for the next cleanup after the current execution is successfully completed or interrupted by failure. This is a repetitive process.

[0082] After the metadata migration and update are completed, the space organization task is successful, and the target object can now provide external services.

[0083] S6, perform cleanup operations and end the task.

[0084] After the layout information is updated, the target object can provide object read, write, and delete services, but the state must be updated first. If the target object is not full, space compaction can continue. If it is full, the space compaction function is notified to release the space, and the state returns to normal. The client can then perform read, write, and delete operations on the target object.

[0085] Specifically, as shown in Figure 7, the background begins to perform the cleanup activities of the space cleanup task, including: updating the status of the target object, cleaning the source object, cleaning the memory structure solidified by the space cleanup, and cleaning the solidified space cleanup task. Then a new round of space cleanup work begins.

[0086] (1) Update the status of the target object: delete the space sorting mark and update the status of the target object to normal.

[0087] (2) Clean up the source object: clean up the metadata and object table of the source object, and destroy the source object.

[0088] (3) Clean up task records: clean up GC memory structure resources and solidification tasks.

[0089] After completing the above steps, the valid data on one or more source objects is centrally migrated to the contiguous space of one or more target large objects. The metadata corresponding to the valid data is also updated simultaneously, ensuring data consistency and integrity. It should be noted that the actual capacity of the target large object is not necessarily the full 96MB. If the size of the last object cannot be fully inserted into the large object space, a certain amount of space is reserved at the end.

[0090] The distributed object storage space organization method of the present invention can be applied to the state transmission of target objects in video surveillance and streaming media data. These applications require not only the transmission of large amounts of media data such as audio and video, but also the recording and analysis of large quantities of small objects such as logs, text, and images. The distributed object storage space organization method of the present invention can improve data storage and space utilization for media storage with a large number of delete and overwrite operations.

[0091] In a cloud storage environment, there are also a large number of objects of different sizes. This application can optimize the storage space in the cloud storage system, improve storage efficiency, and reduce storage costs.

[0092] The distributed object storage space organization method according to the embodiment of the present application has the following beneficial effects:

[0093] (1) Dynamic migration strategy for small objects: By continuously integrating valid data segments in large objects with a large gap between valid space and allocated space into new empty large objects, the purpose of eliminating holes is achieved, while also reducing the impact on system load when performing space sorting. This dynamic adjustment and migration strategy can dynamically adjust the threshold according to cluster capacity usage and system load, and automatically start background space sorting tasks. This optimization strategy allows the system to maintain high efficiency under different loads, and solves the problem of increased storage space fragmentation caused by repeated writing and deletion, or overwriting of massive small objects, resulting in high complexity of metadata management and affecting object reading and writing efficiency.

[0094] (2) Object metadata maintenance method: The metadata and object table maintenance method ensures the consistency and integrity of data and metadata before and after migration by establishing an interval mapping table. By establishing an interval mapping table, data migration is performed while also migrating metadata and object tables, ensuring data consistency and integrity.

[0095] (3) Space consolidation control structure: Adaptive dynamic adjustments are made based on cluster capacity utilization and load, including task concurrency, minimum space threshold, and task execution interval, to improve space consolidation efficiency and data storage continuity. By migrating undeleted small objects from the original large object to a new empty large object, storage space fragmentation and waste are effectively reduced, thereby improving storage space utilization.

[0096] (4) Storage Space Optimization: Existing technologies eliminate storage holes by migrating data within the original large object's own space. This application effectively reduces storage space fragmentation and waste by migrating undeleted small objects from the original large object to a new, empty large object, thereby improving storage space utilization. This application can help users and enterprises use storage space more efficiently, reduce fragmentation, and lower storage costs.

[0097] (5) Adaptive space arrangement control structure: Adaptively adjust various parameters of background tasks, including concurrency, task interval, etc., according to cluster capacity usage and system load.

[0098] (6) Data continuity: Optimizing storage space helps create continuous data storage blocks, thereby reducing random access overhead and speeding up read and write operations.

[0099] (7) Simplified operation and maintenance: The system supports turning on and off background space cleaning tasks at any time. Once turned on, various parameters are adaptively adjusted according to the current situation of the cluster to improve the efficiency of void elimination. No complex operations are required to monitor and manage tasks. Simple query commands are required to view the intermediate process and results.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of this application, and are not intended to limit them. Although this application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein with equivalents; and these modifications or replacements do not deviate from the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of this application. Any changes or replacements that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be covered by the scope of protection of this application.

Claims

1. A method for organizing a distributed object storage space, comprising: S1: Obtain aggregated large objects that meet the hole rate requirement as source objects, and insert the source objects whose fragment space meets the minimum hole threshold into the hash list; S2, selecting the source object with the highest fragmentation rate from the hash list for space arrangement, and creating a target aggregate large object as the target object according to demand; S3, traversing the object table of the source object in batches, obtaining offset information of small objects on the source object, and generating an ordered object table after sorting. According to the ordered object table, obtaining interval information of each small object on the large object, adding the interval information to the interval mapping table of the source object, assigning corresponding target object intervals to the source object intervals, and then generating a target object mapping relationship table; S4, migrating the small object data on the source object to the target object according to the target object mapping relationship table, wherein: First, data migration is performed, including: reading the interval data corresponding to the small objects in the source object, and then writing it to the new interval marked by the target object. After receiving the data write success response, it indicates that the data migration is successful; Then, metadata and object table migration is performed, including: inserting the small object list into the object table LSM tree corresponding to the target object in batches, and inserting the metadata of the source object after interval sorting into the metadata LSM tree of the target object; S5: After the migration is successful, the small objects in the source object are read in batches, and the interval layout information corresponding to the metadata of the small objects themselves is updated according to the interval layout information corresponding to the small objects on the target object; S6, perform cleanup operations and end the task.

2. The distributed object storage space organization method according to claim 1, wherein: The minimum hole threshold is dynamically adjusted according to the cluster capacity utilization rate.

3. The distributed object storage space organization method according to claim 1, wherein: Each source object can only select one target object, and each target object corresponds to multiple source objects.

4. The distributed object storage space organization method according to claim 1, wherein: In S2, space compaction begins, the target object is automatically created in the bucket, and the states of the source object and the target object are set to space compaction states, wherein the sum of the effective capacities of the source objects cannot exceed the capacity of the target object.

5. The distributed object storage space organization method according to claim 1, wherein: In S3, the object table information on the source object is stored in the LSM tree, and the object table and related interval memory on the source object are obtained through the LSM enumeration interface.

6. The distributed object storage space organization method according to claim 1, wherein: In S3, the offset information of the small objects on the source object is obtained, and the offset information of the small objects on the source object is obtained according to the object name and segment number, and the small objects are sorted from large to small to generate an ordered object table.

7. The distributed object storage space organization method according to claim 1, wherein: In S3, after reading the interval mapping table of the source object, the offset information and layout information of the small object are modified to form a continuous address, and the target object mapping relationship table is generated.

8. The distributed object storage space organization method according to claim 1, wherein: In S5, before updating the layout information of the small information object, it is first checked whether the small object still exists. If it does not exist, it means that it has been deleted during the space arrangement, and there is no need to update the layout information.

9. The distributed object storage space organization method according to claim 1, wherein: In S5, when updating the layout information of the small information object, the update is performed in the order of the offset information of the small object, and it is determined whether the update process is normal. If it is normal, the memory is solidified after the update to ensure that the subsequent sorting task can be continued from the failed offset when an exception occurs; if it is an exception, it means that the layout information of some objects is updated successfully, and the source object and the target object both have valid data, which are used as the source object for the next spatial sorting.

10. The distributed object storage space organization method according to claim 1, wherein: In step S6, source objects are cleaned up, and the solidified memory structure and solidified space arrangement tasks are cleaned up.

Images