A snapshot information hierarchical management method for a distributed storage system

Abstract

The application provides a snapshot information hierarchical management method of a distributed storage system. The snapshot information hierarchical management method of the distributed storage system can reduce the frequent update and storage pressure of the header object by storing the snapshot information to multiple clone objects in layers, and effectively improves the performance and data consistency of the distributed storage system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of snapshot information hierarchical management of distributed storage systems, and particularly relates to a snapshot information hierarchical management method of a distributed storage system. BACKGROUND

[0002] As a core technology of modern data storage and management, distributed storage systems are widely used in key fields such as cloud computing, big data processing and edge computing. In related technologies, a storage architecture supporting high concurrency access and data version control is constructed through the collaborative work of object storage devices (OSD), snapshot mechanisms and metadata management. Specifically, this technical system covers the whole process from data writing, snapshot generation to data recovery, including key links such as snapshot information structure (SnapSet), object basic information (OI) and clone object (Clone Object). With the continuous growth of data size, the traditional centralized management of snapshot information gradually exposes performance bottlenecks, and systematic optimization is needed in the storage structure and access mechanism.

[0003] However, in the existing snapshot information management method, a single storage strategy of saving all snapshot information in the header object (Header Object) is directly adopted, and the snapshot information is not processed in layers, which may cause the snapshot ID list (snaps, clones) and interval mapping (clone_overlap) fields to occupy 32kB to hundreds of kB of storage space in extreme cases, thereby causing the problem of rapid increase of KV read-write pressure, system performance decline and even flow interruption. In addition, due to the frequent updating of the header object, if interruption occurs during operation, data inconsistency is easy to cause, affecting the reliability and data recovery efficiency of the system. Therefore, how to reduce the storage pressure and improve the access performance through hierarchical management strategy under the premise of ensuring data consistency has become a key technical challenge to be solved in the current distributed storage system. SUMMARY

[0004] The present application aims to at least solve one of the technical problems in the related art to some extent.

[0005] To this end, a first object of the present application is to provide a snapshot information hierarchical management method of a distributed storage system.

[0006] A second object of the present application is to provide a snapshot information hierarchical management device of a distributed storage system.

[0007] A third object of the present application is to provide an electronic device.

[0008] A fourth object of the present application is to provide a computer readable storage medium.

[0009] A fifth object of the present application is to propose a computer program product.

[0010] To achieve the above object, the first aspect of the present application provides a snapshot information hierarchical management method of a distributed storage system, comprising: S1, judging whether there is an update in a snapshot list carried in a request relative to a snapshot list saved in a current header object; S2, if there is an update, judging to generate a normal clone object or a hierarchical clone object according to an OI information size of the header object, and copying part of snapshot information to a corresponding clone object; S3, cutting snapshot information in the header object, and inserting an identification of a newly generated hierarchical clone object in a hierarchical clone object list; and S4, during a read operation, judging a clone object to which a target data range belongs layer by layer according to a hierarchical clone object list and a clone_overlap field saved in the header object, and obtaining a corresponding positive relation from the clone object to return data.

[0011] In an embodiment of the present application, the judging whether there is an update in the snapshot list carried in the request relative to the snapshot list saved in the current header object further comprises: S11, comparing the snapshot list carried in the request with the snapshot list saved in the header object item by item to determine added, deleted or modified snapshot ids; and S12, if the number of added snapshot ids exceeds a preset threshold, preferentially triggering generation of the hierarchical clone object.

[0012] In an embodiment of the present application, the judging to generate the normal clone object or the hierarchical clone object according to the OI information size of the header object further comprises: S21, when the OI information size of the header object is less than or equal to 4 KB, generating the normal clone object, and copying only a size, a mtime and a snapshot list dependent on the clone object; and S22, when the OI information size of the header object exceeds 4 KB, generating the hierarchical clone object, and copying a snapshot list, a clone object list and a clone_overlap field information.

[0013] In one embodiment of the present application, the read operation is, according to the layered clone object list and the clone_overlap field saved in the header object, to determine the clone object to which the target data range belongs layer by layer, and further comprises: S41, if the clone_overlap field of the current header object does not completely cover the requested data range, then sequentially read the OI information of the next clone object in the layered clone object list; S42, according to the clone_overlap field of each layered clone object, calculate the data range covered thereby, and match it with the requested data range, until a clone object completely covering the requested range is found.

[0014] In one embodiment of the present application, it further comprises: S5, according to the snapshot id and data range requested by the client, pre-load the OI information of the layered clone object that may be involved, so as to reduce the delay of subsequent read operation.

[0015] To achieve the above purpose, the second aspect embodiment of the present application proposes a distributed storage system snapshot information layered management device, comprising: a snapshot update judgment module for judging whether the snapshot list carried in the request is updated relative to the snapshot list saved in the current header object; a clone object generation module for generating a normal clone object or a layered clone object according to the OI information size of the header object if there is an update, and copying part of the snapshot information to the corresponding clone object; a snapshot information clipping and insertion module for clipping the snapshot information in the header object and inserting the newly generated layered clone object identifier into the layered clone object list; a data read judgment module for, in a read operation, judging the clone object to which the target data range belongs layer by layer according to the layered clone object list and the clone_overlap field saved in the header object, and obtaining the corresponding positive relationship therefrom to return data.

[0016] In one embodiment of the present application, the snapshot update judgment module is further used for: comparing the snapshot list carried in the request with the snapshot list saved in the header object item by item to determine the added, deleted or modified snapshot id; if the number of added snapshot ids exceeds a preset threshold, then preferentially trigger the generation of a layered clone object.

[0017] To achieve the above purpose, the third aspect embodiment of the present application proposes an electronic device, comprising: a processor, and a memory in communication connection with the processor;

[0018] The memory stores computer execution instructions;

[0019] The processor executes computer-executed instructions stored in the memory to implement the method of any one of the first aspect.

[0020] To achieve the above object, the fourth aspect of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores computer-executed instructions, and the computer-executed instructions are executed by a processor to implement the method of any one of the first aspect.

[0021] To achieve the above object, the fifth aspect of the present application provides a computer program product, wherein the computer program is executed by a processor to implement the method of any one of the first aspect.

[0022] The embodiments of the present application provide at least the following beneficial effects:

[0023] By layering the snapshot information to multiple clone objects, the frequent update and storage pressure of the header object are reduced, and the performance and data consistency of the distributed storage system are effectively improved.

[0024] Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following and / or by practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0025] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description, taken in conjunction with the accompanying drawings, in which:

[0026] Figure 1 is a flow chart of a snapshot information hierarchical management method of a distributed storage system according to an embodiment of the present application;

[0027] Figure 2 is a flow chart of a header object writing method according to an embodiment of the present application;

[0028] Figure 3 is a flow chart of a header object reading method according to an embodiment of the present application;

[0029] Figure 4 is a flow chart of a clone object reading method according to an embodiment of the present application;

[0030] Figure 5 is a structural schematic diagram of a snapshot information hierarchical management device of a distributed storage system according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] Embodiments of the present application are described below in the context of fully functional methods and systems. Those of ordinary skill in the art will recognize that the implementations described herein can be carried out by appropriate means including, but not limited to, hardware, firmware, software, or combinations thereof. Examples of software can include sensor-actuator control algorithms, decision feedback algorithms, and other algorithms for achieving the results described herein. Examples of firmware can include microcode to control one or more microarchitectural elements of a processor to support the techniques described herein. Examples of hardware can include any or all of the hardware elements described herein, including but not limited to processor elements, controller elements, arithmetic logic unit elements, and the like. Any of the elements of a typical computing device can be used in implementing the described embodiments. The embodiments described below are meant to be exemplary and are intended to be interpreted as limiting only as specified in the claims.

[0032] Embodiment 1

[0033] Figure 1 is a flowchart of a snapshot information hierarchical management method of a distributed storage system according to an embodiment of the present application.

[0034] As shown in Figure 1 , the snapshot information hierarchical management method of a distributed storage system includes the following steps:

[0035] S1, determine whether the snapshot list carried in the request is updated relative to the snapshot list saved in the current header object.

[0036] Specifically, in the distributed storage system, determining whether the snapshot list carried in the request is updated relative to the snapshot list saved in the current Header object is one of the key steps to implement the snapshot information hierarchical management optimization method. The core of this step is to determine whether to trigger the Clone operation of the object by comparing the version information and content difference of the snapshot ID list, so as to decide whether to migrate part of the snapshot information to the hierarchical Clone object, so as to relieve the storage pressure of the Header object and improve the system performance.

[0037] In terms of technical implementation, this determination process is based on the snaps field in the Object_Info_t structure. The snapshot list carried in the request is usually passed in the form of vector<snapid_t>, and the snapshot list saved in the Header object is also stored in this structure. The system first compares the latest snapshot ID (seq) in the request snapshot list with the latest snapshot ID recorded in the Header object. If the request snapshot ID is greater than the ID recorded in the Header, it means that there is an update. Further, the system will compare the two snapshot lists one by one to determine whether there are new, deleted or changed snapshot IDs. This comparison process can be implemented using hash check or difference algorithm (such as binary search based on ordered list) to improve efficiency.

[0038] At the parameter index level, the maximum length of the snapshot list is usually set to 4096 snapshot IDs, each occupying 4 bytes, so the snapshot list can occupy about 32 kB of storage space in the extreme case. The overall size of the OI information of the Header object is limited to 4 KB, and when the snapshot information approaches this threshold, the system will preferentially trigger the hierarchical Clone operation to migrate part of the snapshot information to the hierarchical Clone object. The judgment result of this step directly affects the subsequent Clone strategy selection and is the basis for the system to achieve dynamic load balancing and storage optimization.

[0039] At the application scenario level, this step is widely used in cloud storage, big data platforms, and distributed file systems, especially in scenarios with high concurrent writes and frequent snapshot operations. For example, in distributed storage systems such as Ceph, when a client initiates a snapshot creation or deletion request, the system needs to quickly determine whether the Header object needs to be updated or cloned to ensure data consistency and reduce I / O latency.

[0040] The technical effect of this step is that through the refined snapshot information update judgment mechanism, it effectively avoids the performance bottleneck caused by frequent updates of the Header object. At the same time, this judgment logic provides a decision basis for subsequent hierarchical Clone operations, thereby realizing dynamic hierarchical management of snapshot information and improving the scalability and stability of the system.

[0041] Further, S1 comprises:

[0042] S11, compare the snapshot list carried in the request with the snapshot list saved in the header object item by item to determine the added, deleted, or modified snapshot IDs.

[0043] Specifically, this step involves incremental comparison processing of snapshot information in a distributed storage system, and its technical implementation principle is based on the hierarchical management mechanism of snapshot information. Specifically, when a client initiates a request containing a snapshot list, the system needs to compare the snapshot list with the snapshot information saved in the header object item by item to identify the added, deleted, or modified snapshot IDs. This comparison process is a key link for realizing dynamic update and hierarchical storage of snapshot information.

[0044] In terms of technical implementation, the system first extracts the list of snapshot IDs from the request and compares them one by one with the snaps field in the Object_Info_t structure in the header object. The snaps field is an ordered list of snapshot IDs, usually in descending order, identifying the snapshot set on which the object depends. The comparison algorithm can use double-pointer traversal or hash table acceleration to achieve O(n) or O(1) time complexity for efficient identification. For each snapshot ID, the system determines whether it exists in the snaps list of the header object. If it does not exist, it is marked as new; if it exists but the order or number has changed, it is marked as modified; if it is missing in the request, it is marked as deleted. This process must follow a consistency protocol to ensure atomic update of the snapshot list under concurrent access.

[0045] In terms of parameter indicators, snapshot IDs are usually 64-bit integers (snapid_t), and the maximum length of the snaps list is 4096 elements, corresponding to the upper limit of the number of snapshots. Each snapshot ID occupies 8 bytes in memory, and the list as a whole occupies about 32 kB. To ensure system performance, the total size of the OI information in the header object must be controlled within 4 KB. When the snapshot information approaches this threshold, the system will trigger a hierarchical clone operation to migrate part of the snapshot information to the hierarchical clone object, thereby avoiding frequent updates and excessive storage pressure on the header object.

[0046] This step is widely used in cloud storage, big data platforms, and distributed file systems in practical applications, especially in scenarios with high concurrency and high-frequency snapshot operations, such as virtual machine snapshot management and database version control. By accurately identifying changes in snapshot IDs, the system can dynamically adjust snapshot dependencies, ensuring data consistency and efficient access.

[0047] In terms of technical effects, this step effectively reduces the update frequency and storage pressure of the header object, improving the system's throughput and response speed. At the same time, through the hierarchical management mechanism, the system has good scalability and can support large-scale snapshot operations without affecting overall performance, providing a key guarantee for the stability and efficiency of distributed storage systems.

[0048] S12, if the number of new snapshot IDs exceeds the preset threshold, the generation of the hierarchical clone object is triggered first.

[0049] Specifically, in some implementations, when the number of newly added snapshot IDs (snapids) in the distributed storage system exceeds a preset threshold, the system will preferentially trigger the generation of hierarchical clone objects to optimize the storage structure of snapshot information and improve system performance. The technical implementation of this step is based on the hierarchical management mechanism of snapshot information, and its core principle is to dynamically judge the size of snapshot information and migrate part of the snapshot metadata from the header object to the hierarchical clone object, thereby avoiding the performance bottleneck caused by frequent updates of the header object.

[0050] The specific operation is as follows: after receiving the snapshot creation request, the system first parses the snapshot ID list carried in the request and compares it with the snapshot ID list currently saved in the header object. If the number of newly added snapshot IDs exceeds the preset threshold (for example, the system is set to 128 snapshot IDs), the generation process of the hierarchical clone object is triggered. At this time, the system will judge according to the Object Info (OI) information size of the header object. If the OI size is close to or exceeds 4KB (i.e. the upper limit of KV operation set by the system), a hierarchical clone object is generated instead of a normal clone object. The hierarchical clone object not only contains basic OI information (such as object size, modification time, and dependent snapshot list), but also copies additional key metadata such as snapshot list (snaps), clone object list (clones), and clone_overlap in the header object. Subsequently, the header object performs a trim operation on these information, i.e. deletes redundant or historical snapshot information to keep its OI size within a controllable range, and inserts the ID of the newly generated hierarchical clone object in the clone_list for subsequent chain lookup when reading.

[0051] The key parameters involved in this step include the snapshot ID quantity threshold (such as 128), the OI information size threshold (such as 4KB), the clone object type (normal / hierarchical), etc. By setting reasonable thresholds, the system can timely perform hierarchical processing when the number of snapshots increases, thereby effectively controlling the storage pressure of the header object, reducing the frequency and delay of KV operations, and improving the concurrent processing capability of the system.

[0052] In actual application scenarios, this step is suitable for large-scale distributed storage systems, especially in cloud storage, data centers, and edge computing environments. When snapshots are frequently created for volumes or directories, the update frequency of the header object can be significantly reduced, and performance degradation or even service interruption caused by snapshot information inflation can be avoided. Through the generation of hierarchical clone objects, the system realizes dynamic migration and load balancing of snapshot information, providing efficient data structure support for subsequent snapshot reading and data recovery.

[0053] The technical effect of this step is that by introducing the generation mechanism of hierarchical clone objects, the storage pressure of header objects in snapshot information management is effectively alleviated, the efficiency of KV operations is improved, thereby enhancing the stability and response capability of the distributed storage system in high concurrency and large data scenarios, and having significant innovation and practicality.

[0054] S2, if there is an update, a normal clone object or a hierarchical clone object is generated according to the OI information size of the header object, and part of the snapshot information is copied into the corresponding clone object.

[0055] Specifically, in the distributed storage system, when detecting that the snapshot information exists update, the system will dynamically judge to generate a normal clone object or a hierarchical clone object according to the OI (Object Info) information size of the header object, and copy part of the snapshot information to the corresponding clone object. This step is one of the core mechanisms to realize the optimization of hierarchical management of snapshot information, and its technical implementation is based on the structured storage and dynamic hierarchical strategy of snapshot information in the object storage system.

[0056] In some implementations, the system first compares the snapshot list carried in the current request with the snapshot list already saved in the header object to determine whether there is new or changed snapshot information. If there is an update, the clone operation is triggered. At this time, the system will evaluate the OI information size of the header object. If the size does not exceed the preset threshold (for example, 4KB), a normal clone object is generated, and only the basic attributes in the header object are copied, such as object size (size), modification time (mtime) and snapshot ID list (snaps) dependent on the clone object. If the OI information size has approached or exceeded the threshold, a hierarchical clone object is generated, which not only copies the basic attributes, but also contains snapshot list (snaps), clone object list (clone_list) and clone_overlap and other key snapshot relationship information, thereby realizing hierarchical storage of snapshot information.

[0057] Further, after the header object generates a hierarchical clone object, it will perform a trim operation on its OI information, that is, delete or compress part of the redundant snapshot information to ensure that its own size is always controlled within 4KB. At the same time, the system inserts the identification information of the newly generated hierarchical clone object in the clone_list of the header object, which is used for chain lookup when reading later.

[0058] This step is widely applicable to large-scale distributed storage systems in practical applications, especially in cloud storage, data centers, and edge computing scenarios. When objects are frequently modified and the number of snapshots is large, it can effectively alleviate the read-write pressure of header objects and improve the overall performance of the system. By distributing snapshot information to multiple clone objects, the system can avoid performance bottlenecks caused by snapshot information expansion while ensuring data consistency and version control capabilities, thereby achieving efficient and reliable data management.

[0059] Further, S2 comprises:

[0060] S21, when the OI information size of the header object is less than or equal to 4KB, a normal clone object is generated, and only size, mtime and the snapshot list dependent on the clone object are copied.

[0061] Specifically, when the OI (Object Info) information size of the header object is less than or equal to 4KB, the system will generate a normal clone object and only copy size, mtime and the snapshot list dependent on the clone object. This step is one of the core implementations of the snapshot information hierarchical management optimization method in the present application, aiming to reduce the storage pressure of snapshot information in the header object and improve the performance and stability of the distributed storage system.

[0062] In terms of technical implementation, this step is based on the hierarchical management mechanism of header objects and clone objects in object storage systems. When the header object receives a write request, the system first evaluates the size of the current OI information. If the OI information size does not exceed the 4KB threshold (this threshold is the system's preset KV storage optimization standard, usually related to the performance bottleneck of the underlying storage engine), a normal clone operation is triggered. The generation process of the normal clone object only copies the key metadata fields in the header object, including object size (size), modification timestamp (mtime), and the snapshot ID list (snaps) related to the clone object. These fields together constitute the minimum necessary metadata information of the clone object, which is used for subsequent snapshot reading and version control.

[0063] In terms of parameter indicators, the size field is usually a 32-bit or 64-bit integer, used to record the size of the current data block of the object; mtime is of the utime_t type, recording the timestamp of the last modification of the object, with a precision of microseconds; the snapshot list snaps is of the vector<snapid_t> structure, and snapid_t is a snapshot identifier type, usually a 64-bit integer. In a normal clone object, the snaps field only contains snapshot IDs dependent on the clone, rather than all snapshot information, thereby effectively controlling the size of the OI structure within 4 KB.

[0064] In application scenarios, this step is suitable for scenarios with a small number of snapshots or a low data update frequency. For example, in a cloud storage system, when a user performs a snapshot creation or deletion operation on an object, if the OI information of the header object has not reached the storage bottleneck, the system will isolate the snapshot information by generating a normal clone object to avoid performance degradation caused by frequent updates to the header object. This mechanism is particularly suitable for large-scale distributed storage systems, such as Ceph, HDFS, etc., which rely on KV storage engines (such as LevelDB, RocksDB) for metadata management.

[0065] The technical effect of this step is that by copying only the key metadata fields, the storage overhead of the clone object and the complexity of the KV operation are significantly reduced. When the size of the header object does not reach the 4 KB threshold, the generation of a normal clone object avoids the duplication of redundant snapshot information, thereby reducing storage space occupation and network transmission load, and improving the overall throughput and response speed of the system. At the same time, this mechanism provides a foundation for the generation of subsequent hierarchical clone objects, ensuring that the system performance remains stable when the snapshot information grows.

[0066] S22, when the OI information size of the header object exceeds 4 KB, a hierarchical clone object is generated, and the snapshot list, clone object list, and clone_overlap field information are copied.

[0067] Specifically, when the OI (Object Info) information size of the header object exceeds 4 KB, the system will trigger the operation of generating a hierarchical clone object, which is one of the core implementation steps of the snapshot information hierarchical management optimization method in the present application. The technical principle of this step is based on the dynamic growth characteristics of snapshot information in object storage systems, by separating part of the snapshot metadata from the header object and copying it to an independent hierarchical clone object, thereby effectively controlling the size of a single KV (Key-Value) object and avoiding performance bottlenecks caused by frequent updates.

[0068] In a specific implementation, when the total size of the OI information (including the snapshot ID list, the clone object list, and the clone_overlap field) of the header object exceeds the 4 KB threshold, the system will perform a hierarchical clone operation. This operation not only copies the basic attributes of the header object (such as size, mtime, etc.), but also specifically copies its snapshot list (snaps), clone object list (clones), and clone_overlap field. Among them, the clone_overlap field is a mapping structure used to record the data range corresponding to each snapshot ID and the coverage relationship with the previous clone object, and its data structure is map<snapid_t, interval_set<uint32_t>>. In the extreme case, it may occupy hundreds of KB of storage space. Therefore, when the OI information of the header object is close to or exceeds 4 KB, these key snapshot information must be migrated to the hierarchical clone object to ensure that the size of the header object is always controlled within 4 KB, meeting the performance optimization standards for KV operations in object storage systems.

[0069] In an implementation, the system first serializes and calculates the size of the OI information of the header object. If it exceeds the threshold, a new hierarchical clone object is created, and the snapshot information is copied to the OI structure of the object. At the same time, the header object will perform a trim operation on the original OI information, i.e., delete or compress part of the snapshot information to keep its size within the 4 KB range. In addition, the clone_list field of the header object will add the identifier of the hierarchical clone object for subsequent chain lookup when reading.

[0070] This step is mainly aimed at high snapshot density scenarios in large-scale distributed storage systems, such as cloud storage platforms, enterprise backup systems, or big data analysis platforms. In these environments, objects may be frequently snapped, causing the OI information of the header object to rapidly expand. Through the hierarchical clone mechanism, the system can store snapshot information in a distributed manner, reducing the access pressure on individual KV objects and improving overall I / O performance and system stability.

[0071] From a technical effect point of view, this step effectively solves the performance degradation problem caused by traditional centralized management of snapshot information. By storing snapshot information hierarchically, the system can reduce frequent access to the header object during read and write operations, thereby reducing latency and improving throughput. At the same time, this mechanism ensures the integrity and consistency of snapshot information, supports efficient data version control and recovery operations, and provides key support for the scalability of distributed storage systems.

[0072] S3, trim the snapshot information in the header object and insert the generated layered clone object identifier into the layered clone object list.

[0073] Specifically, in some implementations, trimming the snapshot information in the header object and inserting the generated layered clone object identifier into the layered clone object list is one of the core steps of the snapshot information layered management optimization method in the present application. This step aims to reduce the storage pressure of the header object and improve the performance and scalability of the distributed storage system by dispersing the snapshot information into multiple clone objects.

[0074] In terms of technical implementation, this step first determines whether the size of the OI (Object Info) information of the header object exceeds a preset threshold (e.g., 4KB) during the OI information update process. When the threshold is exceeded, the system will trigger the layered clone operation. In the layered clone operation, the header object will trim part of the snapshot information (such as the snapshot ID list snaps, the clone object list clones, and the clone_overlap interval mapping), i.e., separate these information from the header object and copy them to the newly generated layered clone object. The trimming operation, based on the sparsity and time series characteristics of data access, prioritizes the preservation of the latest snapshot information and migrates historical snapshot information to the layered clone object to reduce the frequent updates of the header object.

[0075] In terms of parameter indicators, the key parameters involved in the trimming operation include snapshot ID (snapid_t), data range interval (interval_set<uint32_t>), and the size threshold of the OI structure (4KB). The trimming strategy can be dynamically adjusted based on the usage frequency of snapshots, timestamp ordering, data coverage range, etc. The insertion operation adds the generated layered clone object identifier to the clone_list field of the header object, ensuring that the system can quickly locate the corresponding layered clone object through this list.

[0076] In terms of application scenarios, this step is applicable to large-scale distributed storage systems, especially when the number of snapshots exceeds 4096, the storage pressure of snapshot information increases significantly, and traditional methods easily lead to performance bottlenecks. Through the layered clone mechanism, the system can distribute the storage of snapshot information, which is suitable for cloud storage, big data platforms, edge computing nodes, and other scenarios that require high requirements for data version control and efficient access.

[0077] At the technical effect level, this step effectively alleviates the storage pressure of the header object, avoiding the problem of system performance degradation caused by frequent updates of snapshot information. Through the introduction of pruning and layered clone objects, the system can maintain low KV operation overhead in read and write operations, thereby significantly improving overall throughput and response efficiency, enhancing the stability and scalability of the distributed storage system.

[0078] S4, during read operation, the layered clone object list and clone_overlap field in the header object are used to determine the clone object to which the target data range belongs, and the corresponding positive relationship is obtained to return the data.

[0079] Specifically, in some implementations, when performing a read operation, the system determines the clone object to which the target data range belongs according to the layered clone object list (clone_list) and clone_overlap field saved in the Header object, and obtains the corresponding positive relationship to return the data. This step is one of the core read mechanisms of the snapshot information layered management optimization method in the present application, and its technical implementation is based on the modeling of the dependency relationship between snapshots and clone objects in the object storage system, aiming to reduce the read and write pressure of the Header object and improve the performance and stability of the system in high concurrency and large data scenarios.

[0080] From the technical implementation point of view, the Header object first parses the data range (such as offset and length) specified in the client request during read operation, and matches the range with the clone_overlap field saved in itself. The clone_overlap field is a map<snapid_t, interval_set<uint32_t>> structure, which is used to record the coverage of each snapshot ID on the current Clone object. The system traverses this field to determine whether the target data range is completely contained in the current Header object or its associated layered Clone object. If not, the layered Clone objects recorded in the clone_list are read in turn, and the above range matching process is repeated until the Clone object covering all data ranges is found or all layered Clone objects are traversed.

[0081] At the parameter index level, the storage structure of the clone_overlap field needs to meet the efficient query requirement. Each interval_set<uint32_t> is implemented by using a bitmap or an interval tree (Interval Tree) to support fast range lookup and merging operation. The clone_list field of the Header object records the IDs of all hierarchical Clone objects, and the maximum length is limited by system design, usually no more than 256, to ensure that the response time of a single read operation is controlled within milliseconds. In addition, the size of the OI information of each hierarchical Clone object is strictly controlled within 4 KB to meet the performance optimization standard of the KV storage system.

[0082] At the application scenario level, this step is widely applicable to scenarios such as cloud storage, big data analysis platforms, virtualization environments, and the like that need to frequently read historical snapshot data. For example, in a cloud storage system, a user may need to read a file version at a certain time point, and the system quickly locates the storage location of the required snapshot data through the hierarchical Clone information of the Header object, thereby reducing frequent access to the Header object and improving the overall I / O throughput.

[0083] The technical effect of this step is that by dispersing snapshot information into multiple hierarchical Clone objects, the read and write load of the Header object is significantly reduced, and the performance bottleneck caused by the expansion of snapshot information in the traditional way is avoided. At the same time, through the precise range matching mechanism of the clone_overlap field, the accuracy and consistency of data reading are ensured, and the reliability and response efficiency of the distributed storage system in a high-concurrency environment are improved.

[0084] The distributed storage system snapshot information hierarchical management method of the embodiment of the application reduces the frequent update and access pressure of the header object by storing snapshot information into multiple clone objects, effectively improving the performance and reliability of the distributed storage system in a high-data-volume and high-concurrency access scenario.

[0085] Further, S4 comprises:

[0086] S41, if the clone_overlap field of the current header object does not completely cover the requested data range, the OI information of the next clone object in the hierarchical clone object list is read in sequence.

[0087] Specifically, in a distributed storage system, when a client initiates a read request, the system needs to determine the storage location of the required data according to the snapshot information. This step, "if the clone_overlap field of the current header object does not completely cover the data range of the request, read the OI information of the next clone object in the layered clone object list in turn", is one of the key links to realize efficient snapshot data reading.

[0088] Technical implementation level: This step is based on the layered clone object mechanism, and by traversing the clone_list field maintained by the header object, the Object_Info_t (OI) information of the layered clone object is accessed in turn. Each layered clone object contains a clone_overlap field, which is stored in the form of map<snapid_t, interval_set<uint32_t>> and is used to describe the data range covered by the clone object and the difference interval with the previous clone object. The system first checks whether the clone_overlap of the current header object completely covers the data range of the client request. If not, the OI information of the next layered clone object is read from the clone_list, and the judgment process is repeated until a complete clone object covering the request range is found or all layered clone objects are traversed.

[0089] Parameter index level: In implementation, each interval_set<uint32_t> of the clone_overlap field records the starting and ending offset of the data block, with a unit of 4KB data block (block). The system compares the request range with the interval set in clone_overlap to judge whether it meets the condition [start, end] ⊆ clone_overlap. If not, the read operation of the next layered clone object is triggered. This mechanism ensures that each read operation only needs to access no more than 4KB of OI data, which meets the performance optimization standard of kv operation of distributed storage systems such as Ceph.

[0090] Application scenario level: This step is suitable for the read scenario of snapshot data in large-scale distributed storage systems, especially in environments where volume snapshots are frequently created and data is frequently modified. For example, in a cloud storage platform, users may create multiple snapshots for the same object, and each read request needs to locate the correct data version according to the snapshot ID and data range. Through the layer-by-layer reading mechanism of layered clone objects, the system can effectively reduce the access pressure of the header object and improve the reading efficiency.

[0091] Technical effect level: This step avoids the performance bottleneck caused by the header object's snapshot information expansion in the traditional way through the hierarchical reading strategy. In the case of 4096 snapshots and the clone_overlap field occupying hundreds of KB, the system can still maintain stable reading performance, significantly improving the scalability and response efficiency of the distributed storage system. In addition, this mechanism supports efficient data version control and consistency guarantee, providing a solid foundation for high-concurrency and low-latency storage services.

[0092] S42, according to the clone_overlap field of each hierarchical clone object, calculate its covered data range, and match it with the requested data range, until the clone object that completely covers the request range is found.

[0093] Specifically, in a distributed storage system, efficient management of snapshot information is a key link to improve system performance and reliability. This step involves calculating the data range covered by each hierarchical clone object according to its clone_overlap field, and matching it with the data range requested by the client, until the clone object that completely covers the request range is found. This step plays a key role in the snapshot reading process, and its technical implementation is based on the structured design of hierarchical clone objects and interval matching algorithms in object storage systems.

[0094] In terms of technical implementation, the Object_Info_t structure of each hierarchical clone object contains the clone_overlap field, which is a map<snapid_t, interval_set<uint32_t>> type, used to record the data range covered by the clone object under a specific snapshot ID. When the client initiates a snapshot reading request, the system first parses the snapshot ID and target data range (usually one or more continuous uint32_t intervals) in the request. Then, the system obtains the hierarchical clone object list clone_list from the header object and reads the OI information of each hierarchical clone object in turn, extracting the clone_overlap interval set corresponding to the request snapshot ID. Through interval matching algorithms (such as interval coverage detection, interval intersection calculation, etc.), the system determines whether the current clone_overlap of the clone object completely covers the requested data range. If not, the next hierarchical clone object is read until the clone object that completely covers the request range is found.

[0095] At the parameter index level, the interval set in the clone_overlap field is usually stored in the interval_set structure, which supports efficient interval merging and query operations. The interval set corresponding to each snapshot ID can support up to 4096 intervals, each interval length is 32-bit integer, supporting up to 2^32 data blocks mapping. In the matching process, the system adopts linear scanning or binary search strategy to reduce the matching time complexity, and ensures that the delay is still low in large-scale snapshot scenarios.

[0096] At the application scenario level, this step is widely used in cloud storage, big data analysis platform and distributed database system, especially in scenarios that need to frequently access historical snapshot data, such as version rollback, data audit, incremental backup, etc. Through the structured management of hierarchical clone objects, the system can effectively reduce the read-write pressure of the header object and avoid performance bottlenecks caused by snapshot information expansion.

[0097] The technical effect of this step is that through the interval coverage matching mechanism of hierarchical clone objects, the system can quickly locate the storage location of the required snapshot data, reduce unnecessary data scanning and IO operations, and significantly improve the snapshot reading efficiency. At the same time, this mechanism supports dynamic expansion and can adapt to the continuous growth of snapshot quantity and data size, ensuring the stable operation of the system in a high-concurrency and large-data environment.

[0098] The snapshot information hierarchical management method of the distributed storage system of the embodiment of the application reduces the frequent update and access pressure of the header object by storing the snapshot information into multiple clone objects, effectively improving the performance and reliability of the distributed storage system in a high-data and high-concurrency access scenario.

[0099] S5, according to the snapshot id and data range requested by the client, preloading the OI information of the hierarchical clone object that may be involved to reduce the delay of subsequent read operations.

[0100] Specifically, this step involves preloading the OI (Object Info) information of the hierarchical clone object that may be involved according to the snapshot ID and data range requested by the client in the distributed storage system to reduce the delay of subsequent read operations. Based on the hierarchical management mechanism of snapshot information, this step optimizes the data access path to improve the response efficiency of the system in a high-concurrency and large-data scenario.

[0101] At the technical implementation level, when the client initiates a read request, the system first parses the snapshot ID and the required data range in the request. Subsequently, the system determines the data segments that the snapshot ID depends on may be distributed in which hierarchical clone objects according to the hierarchical clone object list (clone_list) and the clone_overlap field maintained in the header object. The clone_overlap field is of the type map<snapid_t, interval_set<uint32_t>> and is used to record the coverage relationship between the data range corresponding to each snapshot ID and the previous clone object. By traversing the field, the system can quickly locate the target clone object and preload the OI information thereof into the memory cache, thereby avoiding repeated access to the disk or remote node during actual reading and reducing I / O delay.

[0102] At the parameter index level, the key parameters involved in this step include the snapshot ID (snapid_t), the data range (interval_set<uint32_t>), the OI structure size (which is controlled to be within 4 KB by default), and the hierarchical depth of the clone object. The system preloads the OI information that may be involved in advance to the cache through the preloading strategy, and the cache hit rate can be improved to more than 90%, significantly reducing the number of disk accesses. In addition, the trigger threshold of the preloading operation can be configured, for example, when the matching degree of the data range and the clone_overlap of the current header object is lower than the preset threshold (such as 70%), the system will automatically load the OI information of the next hierarchical clone object.

[0103] At the application scenario level, this step is suitable for scenarios such as cloud storage, big data platforms, virtualization environments, and the like that need to frequently access historical snapshot data. For example, in a cloud backup system, a user may need to restore a data version at a specific time point, at which time the system can quickly locate and return the required data through the preloading mechanism, improving user experience.

[0104] In terms of technical effects, this step effectively reduces the delay of snapshot reading and improves the throughput and response speed of the system by reducing the number of disk I / O and network transmission times. In a large-scale snapshot scenario, the system performance can be improved by more than 30%, while ensuring data consistency and access accuracy. It is one of the core optimization methods of the present application in hierarchical management of snapshot information.

[0105] The distributed storage system snapshot information hierarchical management method of the embodiment of the present application further reduces the response delay of the read operation by preloading the OI information of the related hierarchical clone object according to the snapshot ID and the data range of the client request, and improves the data access efficiency and real-time performance of the system in a high-concurrency scenario.

[0106] Embodiment 2

[0107] Header object write flow with snapshot Figure 2 The processing process first judges whether the snapshot list carried in the request is updated relative to the snapshot list saved in the object OI information. If there is no update, only some basic information of the object OI and the forward relationship of the data are updated; if there is an update, the object clone operation is triggered, and according to the size of the current header object OI, it is judged whether to generate a normal clone object or a layered clone object. For a normal clone object, only some simple OI information of the header object is copied, such as size, mtime and the snapshot list that depends on the clone object. For a layered clone object, the snapshot list, clone object list, clone_overlap information of the header object are also copied, and the header object trims these information and inserts a record in the layered clone object list.

[0108] Header object read flow with snapshot Figure 3 The processing flow first judges whether the data range to be read is contained in the current header object saved clone object according to the clone_overlap in the snapshot information saved by the header object. If it is contained completely, the forward relationship is directly read from the corresponding header object and clone object. If it is not contained completely, the next layered clone object is read, and whether it contains all the remaining data range is judged according to the clone_overlap in the layered clone object. If it is still not complete, the next layered clone object is read, until the data range to be read is found or all layered clone objects are traversed. Finally, the data range corresponding to the target clone object is read according to the found target clone object, and then the data is read according to the forward relationship and returned to the client.

[0109] Read clone object processing process Figure 4 From the above, it can be seen that in the processing flow, first, according to the snapshot id to be read carried in the client request and the layered clone object list saved by the header object, it is judged that the clone object to be read belongs to which layered clone object. Then the OI of the layered clone object is read, and according to the clone_overlap therein, it is judged which clone object the snapshot id to be read belongs to. Then the data segment is calculated on which clone object according to the read data range and clone_overlap. Then the forward relationship is read according to the target clone object and the data segment belonging to it. Finally, the data is read according to the forward relationship and returned to the client.

[0110] In order to achieve the above-mentioned embodiments, the application further provides a snapshot information hierarchical management device of a distributed storage system. Figure 5 A structure schematic diagram of a snapshot information hierarchical management device of a distributed storage system provided by the embodiments of the application is shown in FIG. 1. Figure 5 As shown in the figure, the device comprises:

[0111] A snapshot update judging module 100 is configured to judge whether there is an update in the snapshot list carried in the request relative to the snapshot list saved in the current header object.

[0112] A clone object generating module 200 is configured to, if there is an update, generate a normal clone object or a hierarchical clone object according to the OI information size of the header object, and copy part of the snapshot information to the corresponding clone object.

[0113] A snapshot information clipping and inserting module 300 is configured to clip the snapshot information in the header object, and insert the identification of the newly generated hierarchical clone object in the hierarchical clone object list.

[0114] A data reading judging module 400 is configured to, when reading, judge the clone object to which the target data range belongs layer by layer according to the hierarchical clone object list and the clone_overlap field saved in the header object, and obtain the corresponding positive relation to return data.

[0115] As to the device in the above-mentioned embodiments, the specific way in which each module performs the operation has been described in detail in the embodiments of the method, and will not be described in detail here.

[0116] In order to achieve the above-mentioned embodiments, the application further provides an electronic device, comprising a processor and a memory connected with the processor in communication; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to realize the method provided by the foregoing embodiments.

[0117] In order to achieve the above-mentioned embodiments, the application further provides a computer readable storage medium, wherein the computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to realize the method provided by the foregoing embodiments.

[0118] In order to achieve the above-mentioned embodiments, the application further provides a computer program product, comprising a computer program, and the computer program is executed by the processor to realize the method provided by the foregoing embodiments.

[0119] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the present application comply with relevant laws and regulations and do not violate public order and good customs.

[0120] It should be noted that the personal information from the user should be collected for legal and reasonable purposes, and not shared or sold outside these legal uses. In addition, such collection / sharing should be carried out after the user's informed consent is received, including but not limited to informing the user to read the user agreement / user notice before the user uses the function, and signing the agreement / authorization including authorization of relevant user information. In addition, any necessary steps should be taken to protect and secure access to such personal information data and ensure that other people with access to personal information data comply with their privacy policies and processes.

[0121] The present application is expected to provide embodiments in which the user can selectively prevent the use or access of personal information data. That is, the present disclosure is expected to provide hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, the risk is minimized by limiting data collection and deleting data. In addition, such personal information is de-identified, if applicable, to protect the privacy of the user.

[0122] In the foregoing embodiment descriptions, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples, without contradiction.

[0123] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly specified.

[0124] Any processes or methods described in the flowcharts or otherwise described herein can be understood as representing modules, segments, or portions of code that include one or more executable instructions for implementing specific logic functions (or steps) and / or can be implemented entirely in hardware. The steps or functions of the preferred embodiments of the application can be embodied in software, firmware, or hardware, and can be embodied within an ASIC or other chipset components, which can be specifically adapted to carry out one or more of the processes or methods described herein. Thus, in this or other embodiments, the various steps or functions described can be implemented individually or in sub-combinations or multiples thereof as needed.

[0125] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be embodied in computer executable code that can be translated by a processor-based system into a sequence of instructions to perform a function described by the steps or logic. Thus, in this or other embodiments, the various steps or functions described can be implemented individually or in sub-combinations or multiples thereof as needed.

[0126] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, various steps or methods can be embodied in software or firmware that is stored in memory and executed by a suitable instruction execution system. As such, the software or firmware can be turned into a product that is directly or indirectly tangible if this is deemed necessary by the patent office. If embodied in hardware or in another embodiment, any of the following technologies, or combinations thereof, can be used: a discrete logic circuit having logic gates for implementing logic functions upon an application of data signals; an application specific integrated circuit having appropriate combinational logic gates; a programmable gate array (PGA), such as a field programmable gate array (FPGA), having a memory for storing software; and / or a programmable logic unit (PLU), a microprocessor, or other control unit.

[0127] Those skilled in the art can understand that all or part of the steps of the method of the above-mentioned embodiments can be instructed by a program to complete the relevant hardware, and the program can be stored in a computer readable storage medium. When the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0128] In addition, each functional unit in each embodiment of the present application can be integrated into one processing module, or each unit can exist physically independently, or two or more units can be integrated into one module. The integrated module can be realized in the form of hardware or in the form of a software functional module. When the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0129] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above-mentioned embodiments are exemplary and cannot be understood as limiting the present application. Those skilled in the art can make changes, modifications, replacements and variations to the above-mentioned embodiments within the scope of the present application.

[0130] It should be understood that various forms of flow shown above can be used to reorder, add or delete steps. For example, each step described in the present application can be executed in parallel, sequentially or in different order, as long as the desired results of the technical solutions of the present application can be achieved, which is not limited herein.

[0131] The above detailed description does not constitute a limitation on the scope of protection of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and replacements can be made according to design requirements and other factors. Any modification, equivalent replacement and improvement within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A method for hierarchical management of snapshot information in a distributed storage system, characterized by, Comprise: S1, determine whether the snapshot list carried in the request is updated relative to the snapshot list saved in the current header object; S2, if there is an update, determine to generate a normal clone object or a layered clone object according to the OI information size of the header object, and copy part of the snapshot information to the corresponding clone object; S3, clip the snapshot information in the header object, and insert the newly generated layered clone object identifier into the layered clone object list; S4, during the read operation, according to the layered clone object list and the clone_overlap field saved in the header object, the clone object to which the target data range belongs is determined layer by layer, and the corresponding positive relationship is obtained to return the data; The determination of generating a normal clone object or a layered clone object according to the OI information size of the header object further comprises: S21, when the OI information size of the header object is less than or equal to 4KB, a normal clone object is generated, and only size, mtime and the snapshot list dependent on the clone object are copied; S22, when the OI information size of the header object exceeds 4KB, a layered clone object is generated, and the snapshot list, the clone object list and the clone_overlap field information are copied; The read operation, according to the layered clone object list and the clone_overlap field saved in the header object, further comprises: S41, if the clone_overlap field of the current header object does not completely cover the data range of the request, the OI information of the next clone object in the layered clone object list is read in turn; S42, according to the clone_overlap field of each layered clone object, the data range covered thereby is calculated and matched with the data range of the request until the clone object completely covering the request range is found.

2. The method of claim 1, wherein, The determination of whether the snapshot list carried in the request is updated relative to the snapshot list saved in the current header object further comprises: S11, compare the snapshot list carried in the request with the snapshot list saved in the header object item by item to determine the added, deleted or modified snapshot id; S12, if the number of added snapshot ids exceeds a preset threshold, the generation of a layered clone object is preferentially triggered.

3. The method of claim 1, wherein, Further comprise: S5, according to the snapshot id and data range requested by the client, the OI information of the layered clone object possibly involved is preloaded to reduce the delay of subsequent read operation.

4. A distributed storage system snapshot information hierarchical management apparatus characterized by comprising: Comprise: A snapshot update determination module for determining whether the snapshot list carried in the request is updated relative to the snapshot list saved in the current header object; The clone object generation module is configured to generate a normal clone object or a hierarchical clone object according to the OI information size of the header object, and copy part of the snapshot information to the corresponding clone object if there is an update. The snapshot information clipping and insertion module is configured to clip the snapshot information in the header object, and insert the identifier of the newly generated hierarchical clone object in the hierarchical clone object list. The data reading judgment module is configured to, when reading operation, judge the clone object to which the target data range belongs layer by layer according to the hierarchical clone object list and the clone_overlap field saved in the header object, and obtain the corresponding positive relationship to return data. The clone object generation module is further configured to: generate a normal clone object and only copy the size, mtime and snapshot list dependent on the clone object when the OI information size of the header object is less than or equal to 4KB; generate a hierarchical clone object and copy the snapshot list, the clone object list and the clone_overlap field information when the OI information size of the header object exceeds 4KB. The data reading judgment module is further configured to: read the OI information of the next clone object in the hierarchical clone object list in sequence if the clone_overlap field of the current header object does not completely cover the requested data range; calculate the data range covered by each hierarchical clone object according to the clone_overlap field of the hierarchical clone object, and match the data range with the requested data range until the clone object completely covering the requested range is found.

5. The apparatus of claim 4, wherein, The snapshot update judgment module is further configured to: compare the snapshot list carried in the request with the snapshot list saved in the header object item by item to determine the added, deleted or modified snapshot id; preferentially trigger the generation of a hierarchical clone object if the number of added snapshot ids exceeds a preset threshold.

6. An electronic device, comprising: The processor and the memory connected in communication with the processor; The memory stores computer execution instructions; The processor executes the computer execution instructions stored in the memory to implement the method in any one of claims 1-3. The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the method in any one of claims 1-3.

7. A computer-readable storage medium, characterized in that, The computer program is executed by the processor to implement the method in any one of claims 1-3.

8. A computer program product, characterised in that, ​

Images