A method and system for distributed cache data isolation and recovery based on packet identification

Abstract

The application discloses a kind of based on the distributed cache data isolation and recovery method and system of group identification, the method includes: receiving client object write request, according to object name matching life cycle rule determines effective expiration condition, and generates group identification;With write request, group identification is passed to distributed cache layer. Cache layer is according to object name calculation cache key, when not specified group identification, using default mapping determines cache bucket, when group identification is specified, it is mapped to the group bucket interval isolated from default bucket interval;Further determine cache large object, write client object data, and group identification and the mapping relationship of client object and cache large object are persisted to index. Subsequent read, write, delete and the like request obtains group identification from index, locates the same cache large object, makes the same object set layout in effective expiration condition, and is deleted in life cycle, according to group whole recovery, reduce data void and space stagnation, keep object mapping stable.

Description

Technical Field

[0001] This invention relates to the field of distributed storage technology, and in particular to a method and system for distributed cache data isolation and reclamation based on group identifiers. Background Technology

[0002] Distributed object storage systems typically use objects as the basic unit of data management, distributing objects across multiple storage nodes using object names, hash values, and placement groups. In object storage scenarios, a large number of small files or small objects generate a significant amount of object metadata, index records, and random write requests, thereby increasing the pressure on metadata services, amplifying random I / O, and reducing the efficiency of data recovery and space management.

[0003] To improve the write efficiency of small objects, existing distributed caching layers typically employ an online merging method: after receiving a client's object write request, the business gateway appends the object data to the cached large object and records the mapping relationship between the client object and the cached large object in the index object or index database; once the cached large object reaches its capacity, quantity, or other thresholds, the caching layer flushes it back to the backend storage pool. A typical naming format is: Mcache_{key}_{bucket}_{pos} The key and bucket are calculated based on the client object name and business grouping information to determine the corresponding cached large object for the client object; pos is used to distinguish the sequence number of different cached large objects generated with a fixed capacity under the same key and bucket. Using a fixed key-bucket mapping allows the same client object to locate the same cached large object in subsequent read, write, and delete requests.

[0004] Object storage systems also provide lifecycle management capabilities. Business users can set retention, migration, or deletion rules for objects. For example, objects with a certain name prefix can be deleted a certain number of days after creation, or migrated to low-cost storage media after reaching their retention period. Lifecycle tasks are typically processed in clusters around the same time, handling objects that meet the same rules.

[0005] In existing technologies, online merging schemes for small objects based on object name hashing have been developed. For example, CN202411594313 discloses a method and system for storing massive small files based on indexes. When writing small files, the caching process writes them sequentially to cache objects, records the size and starting position of the small file in the index, and flushes the cache objects back to the backend large object based on the data aging of the cache objects. This type of scheme mainly focuses on small file merging, index recording, and flushing efficiency. However, its key and bucket depend on object name hashing, making it difficult for objects with the same lifecycle or deletion cycle to reliably enter the same large cache object. When client objects with different deletion cycles are appended to the same large cache object, when a batch of objects expires and are deleted, only local data gaps are formed. The large cache object may still not be released in time due to the residue of other objects, resulting in decreased space utilization.

[0006] For lifecycle management, existing solutions also attempt to perform batch processing from a time perspective. For example, CN117349255A discloses a solution based on a lifecycle management index, which builds an index by writing objects per unit time and then deletes expired objects in batches; CN117648297B discloses a solution that merges small objects offline based on write timestamps to reduce gaps in expired deletions. While these solutions consider time factors on the deletion or merging side, their time grouping is mainly based on write time or unit time windows, without establishing a correlation with the object organization method of the cache layer. Therefore, they struggle to solve the problem of gaps caused by the mixing of objects with different deletion cycles in large cached objects.

[0007] Regarding space reclamation, CN119166605A discloses a small file merging space reclamation method based on distributed logs. During overwrite or deletion, it uses an index to determine the affected aggregated objects and ranges, generating a space reclamation log to record the space to be reclaimed. This type of solution can handle space generated by the failure of a single client object. However, when client objects from different deletion cycles are mixed in the same large cached object, even if each client object is reclaimed correctly, a large number of data gaps may still remain for a considerable period, making it difficult to reclaim a batch of objects as a whole according to the same business deletion cycle.

[0008] Regarding grouping and isolation, some caching systems use tenant identifiers, resource group identifiers, key prefixes, or policy-generated identifiers for access isolation, quota allocation, and eviction control. For example, CN115563085B discloses a scheme for generating group identifiers based on policies and binding them to data tables. These identifiers primarily serve access permissions or resource management and are not equivalent to the lifecycle deletion period, nor do they have a consistent association with the object's lifecycle deletion period. Grouping information is difficult to maintain consistency in subsequent operations. If grouping information only exists in the initial write request or a certain processing stage, subsequent operations such as overwrite, deletion, flushing, pass-through, and fault recovery may lose or fail to reuse this information, leading to objects being remapped to incorrect cached large objects or inconsistencies in location between the cache pool and the backend pool.

[0009] Furthermore, in large I / O write-through scenarios, requests may be split into aligned and unaligned parts, written to the backend object and the cached object respectively. If the two parts use different positioning rules, or if the same mapping relationship is not used during flushing, it will further disrupt the consistency of client object positions, increasing the complexity of subsequent reads, deletions, and space reclamation.

[0010] In summary, existing technologies have made progress in areas such as online merging of small objects, batch deletion of lifecycles, space reclamation, and group isolation. However, problems still exist, including mixed storage of objects with different deletion cycles in large cached objects, mismatch between space reclamation granularity and business deletion cycles, lack of correlation between lifecycle rules and cached object organization, difficulty in maintaining consistency of grouping information in subsequent operations, and inconsistencies between write-through and refresh positioning. These shortcomings limit further improvements in small object data isolation, space utilization, and lifecycle reclamation efficiency of the distributed caching layer.

[0011] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0012] The main objective of this invention is to overcome the deficiencies in the aforementioned background technology and provide a method and system for distributed cache data isolation and recycling based on group identifiers.

[0013] To achieve the above objectives, the present invention adopts the following technical solution: A distributed cache data isolation and reclamation method based on group identifiers includes the following steps: S1. Receive a write request from a client object, and determine the valid expiration condition of the client object by matching the lifecycle rules according to the object name; S2. Generate a group identifier based on the valid expiration conditions. The group identifier is used to identify client objects with the same valid expiration conditions. S3. Pass the group identifier along with the write request to the distributed cache layer; S4. The distributed caching layer calculates the cache key based on the object name of the client object and determines the cache bucket based on the group identifier. The calculation of the cache key is independent of whether a group identifier is specified. When no group identifier is specified, the default mapping algorithm is used to determine the cache bucket. When a group identifier is specified, the group identifier is mapped to a group bucket interval isolated from the default bucket interval. S5. Determine the large cache object based on the cache key and the cache bucket, write the data of the client object into the large cache object, and persist the group identifier and the mapping relationship between the client object and the large cache object to the index; S6. In subsequent requests of the client object, obtain the group identifier from the index, and locate the cached large object based on the group identifier, the cache key, and the cache bucket, so that client objects with the same group identifier are grouped into the same cached large object, and are reclaimed as a whole by group when the lifecycle is deleted.

[0014] A distributed cache data isolation and reclamation system based on group identifiers, comprising a business gateway and a distributed cache layer; The business gateway communicates with the client and the distributed cache layer, and is configured to: receive a write request from a client object, match lifecycle rules according to the object name, determine the valid expiration condition of the client object, generate a group identifier according to the valid expiration condition, and pass the group identifier to the distributed cache layer along with the write request; The distributed caching layer is configured as follows: A cache key is calculated based on the object name of the client object, and a cache bucket is determined based on the group identifier, wherein the calculation of the cache key is independent of whether a group identifier is specified; when no group identifier is specified, a default mapping algorithm is used to determine the cache bucket; when a group identifier is specified, the group identifier is mapped to a group bucket interval isolated from the default bucket interval; a large cache object is determined based on the cache key and the cache bucket, the data of the client object is written into the large cache object, and the group identifier and the mapping relationship between the client object and the large cache object are persisted to the index; in subsequent requests of the client object, the group identifier is obtained from the index, and the large cache object is located based on the group identifier, the cache key, and the cache bucket, so that client objects with the same group identifier are grouped into the same large cache object, and are reclaimed as a whole by group when deleted during their lifecycle.

[0015] The present invention has the following beneficial effects: This invention proposes a distributed cache data isolation and reclamation method and system based on group identifiers. It transforms the lifecycle rules of the business gateway into transitive and persistent group identifiers, which are then mapped to cache bucket intervals that do not overlap with the original hash bucket intervals. This allows client objects with the same valid expiration conditions to be stably concentrated in the same large cache object. Compared to existing methods of merging small objects based on object name hashes, objects with different deletion cycles are no longer mixed into the same large cache object. When the same batch of objects expires and is deleted, the mapping relationship on the large cache object can be cleaned up as a whole, thereby releasing cache large object space in a timely manner and reducing local data gaps and space stagnation. Simultaneously, objects without specified group identifiers continue to use the original hash mapping. Group buckets and ordinary buckets are isolated from each other, achieving storage-level isolation without changing the mapping relationship of ungrouped objects, reducing the risk of upgrades and existing data migration.

[0016] This invention persists the group identifier and the mapping relationship between the client object and the cached large object to the index during the initial write. During subsequent reads, overwrites, deletions, flushes, pass-throughs, master-slave synchronization, and fault recovery, the group identifier is retrieved or decrypted from the index, ensuring that the same client object always locates the same cached large object throughout its lifetime. Compared to schemes where the group identifier only exists in the initial write request or a certain processing stage, this invention avoids object location drift caused by the loss or inability to reuse group information in subsequent operations, guaranteeing the stability of the mapping relationship throughout the object's lifecycle.

[0017] In cache flushing and large object write-through scenarios, when large cached objects are flushed back to backend objects, the same cache key, cache bucket, and group identifier are used. Aligned portions are directly written to the backend proxy object, and unaligned portions are also written to the large cached object, sharing the same group identifier, cache key, and cache bucket. This ensures consistent data location between the cache pool and the backend pool. Subsequent reads, overwrites, deletions, and recycling can all be traced back to the corresponding client object and its data range based on the same group identifier, avoiding the problem of inconsistent client object location caused by inconsistent location rules during write-through and flushing.

[0018] Regarding fault recovery and master-slave synchronization, the operation logs, missing tracking information, and object names all retain or can be deciphered from the group identifier, ensuring that master and slave replicas use the same mapping for operations on the same object. The master node can also recalculate the group bucket corresponding to the client object after fault recovery. Compared to solutions where group information is stored in only a single location or cannot be deciphered, this invention reduces the risk of losing group relationships during master-slave synchronization and recovery, and enhances the mapping consistency of the system under abnormal scenarios.

[0019] Furthermore, since objects with the same business rules are more centrally located in the layout, lifecycle batch processing, index traversal, and cache object cleanup can be organized around group buckets, thereby reducing indexing and traversal overhead. These effects are achieved throughout the entire process, from business rule matching, group identifier generation, cache bucket mapping, index persistence to subsequent read / write deletion, flushing, and fault recovery, resulting in an overall improvement in small object data isolation, space utilization, and lifecycle reclamation efficiency for the distributed caching layer.

[0020] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating the overall process of the distributed cache data isolation and reclamation method based on group identifiers according to the present invention.

[0022] Figure 2 This is a schematic diagram of the cache object mapping logic in an embodiment of the present invention.

[0023] Figure 3 This is a flowchart of the first write process in an embodiment of the present invention.

[0024] Figure 4 This is a flowchart illustrating the read / write process of an embodiment of the present invention.

[0025] Figure 5 This is a flowchart illustrating the deletion and space reclamation process according to an embodiment of the present invention.

[0026] Figure 6 This is a flowchart illustrating the cache flushing process in an embodiment of the present invention.

[0027] Figure 7 This is a flowchart illustrating the large object pass-through process according to an embodiment of the present invention. Detailed Implementation

[0028] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.

[0029] This invention aims to overcome the shortcomings of traditional distributed cache small object merging schemes, such as mixed arrangement of objects with different deletion cycles, mismatch between space reclamation granularity and business deletion cycle, and difficulty in maintaining consistency of grouping information in subsequent operations. It proposes a distributed cache data isolation and reclamation method and system based on group identifiers. This method transforms the lifecycle rules of the business gateway into persistent group identifiers, and determines the storage layout of client objects in the cache layer and their subsequent mapping relationships accordingly. This achieves the following: generating verifiable and persistent group identifiers for client objects based on the lifecycle rules of the business gateway; mapping the group identifiers to cache bucket intervals that do not overlap with the original hash bucket intervals, achieving storage-level isolation without changing the mapping relationship of ungrouped objects; maintaining consistency between the group identifier and object mapping relationship during initial write, subsequent read / write deletion, cache flushing, pass-through, master-slave synchronization, and fault recovery; and grouping client objects with the same expiration time into the same large cache object, promptly releasing the large cache object space after the same batch of objects is cleaned up, reducing data gaps.

[0030] See Figures 1 to 7 This invention provides a method for distributed cache data isolation and reclamation based on group identifiers, comprising the following steps: Step S1: Receive the write request from the client object, match the lifecycle rules according to the object name, and determine the valid expiration condition of the client object.

[0031] In some embodiments, the step S1 of determining the valid expiration condition of the client object by matching lifecycle rules based on the object name includes: loading lifecycle rules in units of buckets, wherein the lifecycle rules include an object name prefix and a time condition; inserting the object name prefix character by character into the prefix tree, matching character by character from the root node during matching, and when multiple prefix rules are matched simultaneously, taking the one with the smallest valid deletion period as the matching result; and determining the expiration time of the client object based on the matching result.

[0032] In some embodiments, determining the valid expiration condition of the client object based on the lifecycle rule matching the object name further includes: when the time condition of the lifecycle rule is in minutes, it is rounded up to the nearest day to convert it into a retention period, and the valid expiration time is calculated in combination with the client object creation time; when the time condition is expressed as an absolute date, the time corresponding to the absolute date is directly used as the valid expiration time, no longer depending on the client object creation time; when the same object matches both migration and deletion actions, the deletion condition actually used to clean up the object is used as the valid expiration condition, and the condition used only for storage class migration is not involved in the determination of the valid expiration condition.

[0033] Step S2: Generate a group identifier based on the valid expiration conditions. The group identifier is used to identify client objects with the same valid expiration conditions.

[0034] In some embodiments, the step S2 of generating a group identifier based on the valid expiration condition includes: normalizing the expiration time by day to obtain a normalized date; encoding the normalized date as a group identifier so that objects with valid expiration times falling on the same day generate the same group identifier.

[0035] In some embodiments, generating a group identifier based on the valid expiration condition further includes: rounding down the valid expiration time to a date sequence number, taking the sequence number of that date in the current year as the group identifier, and uniformly using the UTC time zone for calculation, so that objects with valid expiration times on the same day receive the same group identifier.

[0036] Step S3: Pass the group identifier along with the write request to the distributed cache layer.

[0037] In some embodiments, the step S3 of passing the group identifier along with the write request to the distributed cache layer includes: the business gateway sending the group identifier as an additional identifier in the combined object operation, forming an indivisible combined request with the object creation or write data operation, to the distributed cache layer; the additional identifier is carried through CLS class method calls, extended request fields, object attributes, or equivalent mechanisms. Upon receiving the group identifier, the distributed cache layer verifies it. An empty identifier indicates that no group has been specified and is processed according to the default group identifier; an invalid identifier returns a parameter error without silently rewriting it to a valid group. When the grouping function is disabled, newly written objects ignore the group identifier and use the default mapping. When in an upgrade compatibility phase or when group bucket resources reach their limit, new group requests are downgraded to the default mapping; established group buckets continue to be accessed according to the persisted group identifier.

[0038] Step S4: The distributed caching layer calculates the cache key based on the object name of the client object and determines the cache bucket based on the group identifier. The calculation of the cache key is independent of whether a group identifier is specified. When no group identifier is specified, the default mapping algorithm is used to determine the cache bucket. When a group identifier is specified, the group identifier is mapped to a group bucket interval that is isolated from the default bucket interval.

[0039] In some embodiments, such as Figure 2The cache object mapping logic link shown includes the following steps in step S4: calculating the cache key and determining the cache bucket. The cache key is obtained by performing a modulo operation between the hash value of the object name and the cache key space parameter, and the calculation of the cache key is independent of whether a group identifier is specified. When no group identifier is specified, the hash value of the object name is modulo the number of ordinary buckets to obtain the default bucket number. When a group identifier is specified, the number of ordinary buckets is added to the group identifier to obtain the group bucket number, so that the group bucket number falls within an independent interval that does not overlap with the interval of ordinary bucket numbers.

[0040] Step S5: Determine the large cache object based on the cache key and the cache bucket, write the data of the client object into the large cache object, and persist the group identifier and the mapping relationship between the client object and the large cache object to the index.

[0041] In some embodiments, such as Figure 3 The initial write process shown includes step S5, which involves writing the client object's data into the cached large object and persisting the index. During the initial write, the business gateway sends a combined request to the distributed cache layer, consisting of setting a group identifier and creating an object or writing data. The distributed cache layer verifies the group identifier, calculates the cache key and cache bucket, and creates or selects the corresponding cached large object. The client object data is written to the cached large object in an append-only manner, and the mapping information of the client object, cached large object, write location, and group identifier is persisted in the client index. If the write operation fails, the index is not updated, ensuring that the mapping relationship between the group identifier and the object is reliably established.

[0042] In some embodiments, such as Figure 6 The cache flushing process shown includes a cache flushing step after writing the client object's data to the cache large object in step S5: when the cache large object reaches a preset capacity or writing threshold, a flush is triggered; a corresponding backend object is created or located according to the cache key, cache bucket, and location range of the original cache large object; the data in the cache large object is written to the backend object, and the mapping relationship between the client object and the backend object is updated synchronously; the group identifier is retained in the flushing log or object metadata, or the group identifier is deduced from the bucket identifier in the backend object name; after a successful flush, the corresponding cache object in the cache pool is released.

[0043] In some embodiments, the persistence of the group identifier and mapping relationship to the index in step S5 includes: storing the group identifier, cache bucket, and location of the client object in the client index.

[0044] The method also includes steps for synchronizing, restoring, and ensuring compatibility of group identifiers: carrying the group identifier in operation logs related to write, overwrite, delete, and flush operations, so that master and slave replicas use the same mapping for operations on the same object; saving the group identifier in the missing tracking information during fault recovery, and the master node recalculates the group bucket corresponding to the client object based on this; for objects that already exist but lack index caching, retrieving the group identifier from the bucket identifier in the object name; and when upgrading a version or when group bucket resources reach their limit, downgrading new group requests to the default mapping, while existing group buckets continue to be accessed.

[0045] Step S6: In subsequent requests of the client object, obtain the group identifier from the index, and locate the cached large object based on the group identifier, the cache key, and the cache bucket, so that client objects with the same group identifier are grouped into the same cached large object, and are reclaimed as a whole by group when the lifecycle is deleted.

[0046] In some embodiments, such as Figure 4 The read / write process shown in step S6 includes read requests and overwrite requests. The step of obtaining the group identifier from the index and locating the cached large object includes: when the master node processes a read request or an overwrite request, it first queries the client index; if a saved group identifier exists in the index, the group identifier is used to recalculate the cache bucket, and the cached large object recorded in the index and the client object data range are combined to read or update the data; if the group identifier carried in the request is inconsistent with the group identifier saved in the index, the group identifier saved in the index shall prevail.

[0047] In some embodiments, such as Figure 5 The deletion and space reclamation process shown includes the following steps in step S6: During lifecycle deletion, the overall reclamation by group includes: the business gateway generates batch deletion requests for expired objects based on lifecycle rules; each deletion request locates the corresponding cached large object in the distributed cache layer according to the group identifier stored in the index, and clears the mapping relationship between client objects and cached large objects; if there are still mapping relationships between other client objects on the cached large object, only the index relationship of the current object is cleared while the cached large object is retained; if all mapping relationships on the cached large object have been cleared, the cached large object is deleted or its space is fully released; for client objects that have been flushed back to the backend object, the index relationship is cleared according to the same rules until the backend object has no remaining mapping relationships, and then the space is released.

[0048] In some embodiments, such as Figure 7 The large object write-through process shown includes a large object write-through step in the distributed cache data isolation and reclamation method based on group identifiers: For larger I / O requests, split them into aligned and unaligned parts; The aligned portion is written directly to the backend proxy object, and the unaligned portion is written to the cached large object in the cache pool; The aligned portion and the unaligned portion share the same group identifier, cache key, and cache bucket, and the logical range and object name of the two portions are recorded in the metadata. Subsequent reads, overwrites, deletions, and recycling are processed according to the same group identifier.

[0049] In some embodiments, writing the unaligned portion into a large cache object in the cache pool, and flushing it back to the backend proxy object after the large cache object is full, specifically includes: The backend proxy object does not store an index separately. The client index records the offset and length of the client object, the position and offset of the data segment in the backend proxy object, and a status field used to identify whether the data segment belongs to the cache pool or the backend pool for each data segment. Once the cached large object is full, the unaligned data within it is flushed back to the corresponding backend proxy object. A backend proxy object can only carry unaligned data segments of multiple client objects under the same cache key and the same group identifier. During subsequent reads, overwrites, deletions, and recycling, the system traces back to the client object and its data range to which the data segment belongs, based on the status field and group identifier in the client index.

[0050] This invention also provides a distributed cache data isolation and recycling system based on group identifiers, including a business gateway and a distributed cache layer.

[0051] The business gateway communicates with the client and the distributed cache layer, and is configured to: receive a write request from a client object, match lifecycle rules according to the object name, determine the valid expiration condition of the client object, generate a group identifier according to the valid expiration condition, and pass the group identifier to the distributed cache layer along with the write request; The distributed caching layer is configured as follows: A cache key is calculated based on the object name of the client object, and a cache bucket is determined based on the group identifier, wherein the calculation of the cache key is independent of whether a group identifier is specified; when no group identifier is specified, a default mapping algorithm is used to determine the cache bucket; when a group identifier is specified, the group identifier is mapped to a group bucket interval isolated from the default bucket interval; a large cache object is determined based on the cache key and the cache bucket, the data of the client object is written into the large cache object, and the group identifier and the mapping relationship between the client object and the large cache object are persisted to the index; in subsequent requests of the client object, the group identifier is obtained from the index, and the large cache object is located based on the group identifier, the cache key, and the cache bucket, so that client objects with the same group identifier are grouped into the same large cache object, and are reclaimed as a whole by group when deleted during their lifecycle.

[0052] The distributed cache data isolation and recycling method and system proposed in this invention transforms the lifecycle rules on the business gateway side into persistent group identifiers, and uses them to determine the bucket range affiliation of client objects in the cache layer and the mapping of large cache objects. The group identifier is persisted to the index from the first write and is used in subsequent read, write, delete, flush, pass-through and fault recovery, so that objects with the same expiration time are centrally laid out and space is released as a whole during batch cleanup. Thus, while keeping the original mapping of ungrouped objects unchanged, data gaps and space retention are reduced, and the mapping is kept stable throughout the object's lifecycle.

[0053] The following further describes the implementation methods and advantages of specific embodiments of the present invention.

[0054] Abbreviation Definition RGW: RADOS Gateway, an example of an object storage service gateway.

[0055] RADOS: Ceph's distributed object storage layer.

[0056] OSD: Object Storage Daemon, is a storage service process responsible for object data, indexing, and operation processing.

[0057] PG: Placement Group, a group used to organize objects and replicas in a distributed storage system.

[0058] LC: Lifecycle, a lifecycle rule used to determine migration, deletion, and other actions based on object name prefixes, object creation time, or other business conditions.

[0059] `group_id`: Group identifier, used to indicate the business rule group to which the client object belongs, and participates in cache object location. The default value is 0 if no group identifier is specified.

[0060] key: The cache key calculated from the client object identifier, used to locate the key space to which the client object belongs.

[0061] bucket: The bucket identifier for further locating large cached objects under the same key.

[0062] pos: The sequence number of the cached large object; it increments after the cached large object reaches a fixed capacity, and is unrelated to the key-bucket mapping between client objects and cached objects.

[0063] CLS Invocation: A method in which the client passes additional processing information to the backend through class method markers carried by object operations.

[0064] ClientIndex: An index record that tracks the mapping between client objects and cached large objects.

[0065] Mcache: A cached object used for merging small files / objects during writes, prefixed with Mcache_.

[0066] Ucache: A cache object used when writing and merging unaligned portions of large files / objects. Its name is prefixed with Ucache_.

[0067] Backend: Backend objects generated after the cache is full and flushed back; their names are prefixed with Backend_.

[0068] Proxy: The backend object used for large file / object alignment pass-through and Ucache flushing, with the name prefix Proxy_.

[0069] This invention adds a group identifier transmission mechanism between the business gateway and the distributed caching layer. The business gateway matches the object name with lifecycle rules to obtain the object's valid lifecycle conditions and generates a group_id accordingly. When a client object is first written, the business gateway sends the group_id along with the write operation to the distributed caching layer.

[0070] The distributed caching layer simultaneously calculates the key and bucket of the client object: If no group identifier is specified, the original object name hash mapping will be used, and group_id=0; When specifying a group identifier, group_id is converted into a group bucket offset, so that the group bucket falls within an independent bucket range; thus, the algorithm for mapping client objects to cached large objects is no longer limited to object name hashing, but can map client objects to the required cached large objects through group identifiers, realizing custom grouping of objects; For the same client object, the group_id formed by the first write is written to the index and fixed. Subsequent requests read from the index and no longer rely on the client to carry it repeatedly.

[0071] In one implementation, let the number of ordinary buckets be B and the effective group identifier be g, then the grouped buckets can be calculated using the following formula: bucket_group = B + g, g > 0 Regular buckets remain within the original range 0 ... B-1. Therefore, grouped buckets and regular buckets do not overlap. When B is 2, bucket=2 can be used as a reserved interval, g=1 corresponds to bucket=3, and g=2 corresponds to bucket=4. Whether to reserve an interval and the size of the interval can be determined by the specific implementation.

[0072] System Architecture The system includes: Business access module: Receives the first write, read, overwrite, delete and pass-through requests of client objects, and obtains the object name, bucket identifier, object creation time and business rule information.

[0073] Lifecycle rule matching module: Loads lifecycle rules by bucket, matches the object name prefix in the rule with the client object name, and outputs valid lifecycle conditions.

[0074] Group identifier generation module: Calculates or looks up the group_id based on the valid lifecycle conditions. Objects with the same valid expiration time or the same business deletion cycle generate the same group_id.

[0075] The request identifier passing module passes the `group_id` as an additional identifier in the composite object operation to the distributed caching layer. This additional identifier can be carried through CLS class method calls, extended request fields, object properties, or equivalent mechanisms.

[0076] Cache mapping module: Calculates the key, bucket, and cached large object name based on the client object name, storage pool parameters, and group_id.

[0077] Index and ID persistence module: Stores the mapping from client objects to cached objects, the write location, and the group_id in ClientIndex or an equivalent index, and ensures that the ID is not overwritten during the object's lifetime.

[0078] The cache aggregation and flushing module writes multiple client objects to a large cache object in an append-only manner, and flushes the data back to the backend object after the large cache object reaches its capacity threshold.

[0079] The deletion and space reclamation module handles client object deletion and batch deletion of lifecycle objects; cleans up index relationships, and deletes large cached objects or releases their space when there are no remaining mapping relationships for large cached objects.

[0080] Logs and recovery module: The group_id is carried or decrypted in the operation log, missing tracking information and recovery process to ensure that the mapping between the master and slave replicas and after failure recovery is consistent.

[0081] The cache object mapping logic chain is as follows Figure 2 As shown.

[0082] Generation of lifecycle rules to group_id Rule loading The lifecycle matching module caches lifecycle rules in buckets. When processing an object in a bucket for the first time, it reads the rules from the bucket properties or lifecycle configuration store and builds a rule index; after the cache expires, it rereads the rules. Lifecycle rules must include at least an object name prefix and a time condition, which can be days, minutes, or an equivalent parameter that can be converted into a valid deletion period.

[0083] object name prefix matching The object name prefix of each rule is inserted character by character into the prefix tree, and the valid deletion period of the rule is recorded at the node where the prefix ends. Empty prefix rules are recorded at the root node, indicating a match for all objects in that bucket. When processing object names, matching proceeds character by character from the root node, and a prefix rule is only matched if the object name begins with that prefix. When multiple prefix rules match simultaneously, the rule with the smallest valid deletion period (i.e., the one with the earliest valid expiration time) wins, resulting in a unique outcome. If the valid deletion periods are the same, the encoded expiration date and group identifier are also the same, eliminating the need to differentiate priorities. The object name prefix is ​​a literal prefix, and wildcards are treated as ordinary characters.

[0084] Period conversion The time condition for a lifecycle rule can be days, minutes, or an absolute date. For ease of standardized coding, we first convert each time condition into the object's valid expiration time t_expire: d = Days, t_expire = t_create + d × 86400 d = ceil(Minutes / 1440) = (Minutes + 1439) / 1440, t_expire = t_create + d × 86400 t_expire = the time point corresponding to the absolute date When expressed in days, d is directly taken as Days; when expressed in minutes, the retention period d is rounded up by the number of days, for example, 1 minute and 1440 minutes are both converted to 1 day, and 1441 minutes are converted to 2 days; when expressed in absolute dates, t_expire is directly taken as the time point corresponding to the date (e.g., 2024-12-31 00:00:00 UTC), without needing to be converted to a retention period, and no longer depending on the object creation time. t_create is the client object creation time, and 86400 is the number of seconds in a day.

[0085] For rules that include multiple actions such as migration and deletion for the same object, the group identifier should be assigned to the deletion condition actually used to clean up the object; conditions used only for storage class migration are not included in the deletion group.

[0086] Group Identifier Code The group identifier is a deterministic code for the valid expiration date. To ensure that objects with the same expiration date are grouped together, the valid expiration times are first normalized, and then the normalization result is encoded as the group identifier, calculated using the following formula: D = normalize(t_expire) = floor(t_expire / 86400) group_id = encode(D) = day_of_year(D) The `normalize()` function is defined as rounding down to the nearest whole day, reducing the valid expiration time to its corresponding date sequence number, rather than using a fixed recycling batch window for normalization. The `encode()` function retrieves the sequence number of that date within the current year. Therefore, as long as the valid expiration times fall on the same day, the `D` and group identifier are identical, regardless of when the object was created on that day. The calculations uniformly use the UTC time zone to avoid the same object receiving different group identifiers on different nodes when deployed across time zones.

[0087] The value range of `day_of_year()` is 1 to 366 (366 for leap years, 365 for common years). The encoding directly uses the date sequence number as the group identifier without any additional mapping; therefore, the group identifier naturally falls within the valid encoding range of 1 to 65535. The group identifier is reused cyclically yearly, meaning that objects with the same date sequence number but different years will receive the same group identifier. Since the group identifier is only used to group objects in the same recycling batch together, reuse across years does not affect the correctness of isolation and recycling; the specific recycling batch is still based on the actual expiration time during execution. (When there are no matching rules, lifecycle functionality is disabled, or the rule is invalid, `group_id` is 0, indicating the use of the default mapping.)

[0088] Caching object mapping algorithm key calculation The symbols defined in this section are as follows: name represents the client object name; H represents the deterministic hash function; M represents the MCache key space multiple; N represents the original number of keys; B represents the number of ordinary buckets; g represents the valid group identifier; h_b represents the object name or bucket name hash value used for ordinary bucket calculation; key and bucket represent the cache key and cache bucket number, respectively.

[0089] Let the client object name be name, the MCache key space parameter be M, the original number of keys be N, and the hash function be H, then: key = H(name) mod (M × N) This calculation is independent of whether or not the group_id is carried, thus ensuring that the grouping function does not change the basic distribution of the original key space.

[0090] bucket calculation Let the number of ordinary buckets be B, and the hash value of the ordinary bucket corresponding to the object name be h_b: bucket_normal = h_b mod B bucket_group = B + g, g > 0 bucket = bucket_group, g > 0 bucket = bucket_normal, g = 0 Grouped buckets do not use the hash results of regular buckets; instead, they use group identifiers to form deterministic offset buckets. After specifying a group identifier, the mapping algorithm prioritizes calculating the bucket_id according to the group identifier scheme, ensuring that it does not duplicate the original default bucket_id and guaranteeing isolation between different mapping algorithms. In this way, objects with the same group_id under the same key enter the same group bucket, while objects with different group_ids enter different group buckets; ungrouped objects continue to enter regular buckets.

[0091] Cache object naming Cache objects can use the following naming format: Mcache_{key}_{bucket}_{pos} Flush objects can use the same key, bucket, and pos, only replacing the object category prefix, for example: Backend_{key}_{bucket}_{pos} In a passthrough scenario, aligned data uses the Proxy_ backend object prefix, while unaligned data uses the Ucache_ cache object prefix; both parts continue to use the same key, bucket, and group_id.

[0092] First write process The client submits a new small file or object write request to the business gateway.

[0093] The business gateway obtains the object name, bucket identifier, object creation time, and lifecycle configuration.

[0094] The lifecycle rule matching module finds the matching rule through the prefix tree and calculates the valid expiration conditions.

[0095] The group identifier generation module generates group_id; if there is no matching rule, it generates 0.

[0096] The business gateway sends the operation of setting the group identifier and the operation of creating an object / writing data as an indivisible combined request to the distributed caching layer. The combined request can be bound in a single OSD operation through a CLS call, or the atomic execution of setting the group identifier, writing data, and updating the index can be achieved with the help of PG transactions.

[0097] Since this is the first write, the index corresponding to the client object has not yet been established; the cache layer verifies the group identifier, calculates the key and bucket, and creates or selects a large cache object.

[0098] The caching layer appends client object data to the large cache object and persists mapping information such as client object, cache object, length, and group_id in ClientIndex, ensuring successful metadata updates. The master node persists and fixes the group identifier as object metadata, preventing it from changing during the object's lifetime and ensuring the mapping relationship is unique.

[0099] If a write operation fails, the metadata will not be updated, and therefore the group identifier contained in the metadata will not be persisted, ensuring the reliable establishment of the group identifier and object mapping.

[0100] After the master and slave replicas complete data and index synchronization, they return a write success response.

[0101] The initial write process is as follows: Figure 3 As shown.

[0102] Group identifier verification, degradation, and resource limits In one Ceph implementation, the backend performs the following processing on the packet identifier passed in via the request: An empty identifier indicates that no group has been specified, and it will be treated as group_id=0.

[0103] Non-numeric strings, strings with a value of 0, and strings exceeding the range of representation are considered invalid requests and return a parameter error; invalid input should not be silently rewritten as valid groups.

[0104] When the OSD-side grouping function is turned off, newly written objects ignore the group identifier and use the default mapping; objects that have already been written still calculate the mapping using the persistent group_id in the index, which does not affect their subsequent read and write operations.

[0105] During the upgrade compatibility phase or when group bucket resources reach their limit, new group requests can be downgraded to the default mapping; existing group buckets can still be accessed to avoid disrupting established mapping relationships.

[0106] The group bucket limit counts the number of different group buckets created under the same key; regular hash buckets are not included in this limit. An example limit is Ng=128, but the specific value is determined by the configuration.

[0107] Read requests and overwrite requests For read requests, write overwrite requests, or other subsequent operations on client objects, the master node first queries ClientIndex. If a saved group_id exists in the index, it directly uses this identifier to recalculate the cached object name and reads or updates the data based on the cached object and client object data range recorded in the index.

[0108] If the group identifier carried in the request does not match the identifier stored in the index, the identifier in the index has the highest priority; if necessary, the corresponding cached large object will still be created or selected according to the group_id in the index. This ensures that the cached object location of the same client object will not change due to repeated requests, changes in client parameters, or updates to rule configurations during its lifetime.

[0109] Since the lifecycle is based on object name prefix matching, when a user modifies the lifecycle, the object name will change. Consequently, the backend will reassign a new group identifier to create a new object and re-establish the cache mapping, preventing the same object from generating multiple different group identifiers.

[0110] Read request and overwrite process as follows Figure 4 As shown.

[0111] Lifecycle deletion and space reclamation The business gateway generates batch deletion requests for expired objects based on lifecycle rules at similar times. For each deletion request, the cache layer locates the corresponding cached object by the group_id stored in the index and cleans up the mapping relationship between client objects and cached objects.

[0112] If the cached large object still has other client object mapping relationships, only the index relationship of the current object is cleaned up, the records of the corresponding data range are discarded, and the entire cached large object is not deleted; If all mappings on the cached large object have been cleared, then delete the cached large object or perform a full space release on it; For objects that have already been flushed back to the backend pool, the metadata index can still correctly record the relationship between the client object and the backend object. Therefore, the deletion rules are the same: prioritize cleaning up only the index relationship of the object itself, and release space only after the backend object has no other index relationship. Since objects with the same valid expiration time use the same group_id and are mapped to the same group bucket, these objects can be processed centrally in the same recycling batch, reducing data gaps caused by long-term residual objects.

[0113] Since the objects are expiring in the same batch, have the same group identifier, and the cached objects mapped by the client objects are also the same, all small objects on the cached objects will be deleted at the same time. After all the small objects are cleaned up at the same time, the cached objects themselves are deleted in a timely manner to avoid data gaps from existing for a long time and improve space utilization.

[0114] Deletion and space reclamation process as follows Figure 5 As shown.

[0115] Cache refresh process Large objects are cached with a preset capacity or write threshold. When multiple client objects in the same bucket append to and fill the cache with large objects, the cache layer triggers a flush. Lock the large cached objects to be flushed and their index relationships.

[0116] Create or locate the corresponding Backend object based on the key, bucket, and location range of the original cache object.

[0117] Write the data in the cached large object to the backend object and synchronously update the backend location mapping of the client object.

[0118] Retain the group_id in the flush logs or object metadata, and if necessary, extract the group identifier from the bucket in the backend object name.

[0119] After a successful flush, the corresponding cached object in the cache pool is released; the mapping relationship can still be located based on the same key and bucket.

[0120] Because cached objects and backend objects use the same key, bucket, and position relationships, flushing will not change the logical mapping of client objects. Objects grouped with the same expiration time will still maintain a centralized layout on the backend.

[0121] After the data is flushed back to the backend pool, the rules of mapping the same group identifier to the same backend object are still maintained. Therefore, objects with the same LC rule are still on the same large backend object, and data gaps can still be avoided when deletion is triggered.

[0122] The cache refresh process is as follows Figure 6 As shown.

[0123] pass-through process For larger I / O operations that bypass the small object cache merging logic, the business gateway or caching layer will split the request into aligned and unaligned parts: The aligned portion is directly written to the Proxy object in the backend storage; The unaligned portion is written to the Ucache object in the cache pool. Once the Ucache is full, it is flushed back to the corresponding Proxy object. Both parts share the group_id, key, and bucket corresponding to the client object; In the pass-through scenario, the mapping algorithm itself remains unchanged. It uses the group identifier specified by the client to complete the conversion from the client object to the backend object / Ucache object, which is consistent with the algorithm of the cached object. Only the object name prefix changes. Record the logical range and object name of the two parts in the metadata, and process them according to the same group identifier when reading, overwriting, deleting and recycling.

[0124] In pass-through scenarios, the backend Proxy object is also included in the space reclamation based on the group identifier. The Proxy object itself does not store an index separately, but the range of client objects corresponding to each segment of data is recorded in the index of the client object: each mapping in the index records the offset and length of the client object, as well as the position and offset of the data segment in the backend Proxy object. The index contains a state field indicating whether the data segment belongs to the cache pool or the backend pool. Therefore, any segment of data can be traced back to its corresponding client object and its range. Furthermore, since the naming of the Proxy object also includes buckets determined by the group identifier, a Proxy object only carries unaligned data segments of multiple client objects under the same key and the same group identifier.

[0125] Large object pass-through process as follows Figure 7 As shown, the details of the perspective scene are displayed.

[0126] Persistence of group identifiers, master-slave synchronization and fault recovery To prevent the group identifier from being lost in abnormal scenarios, this invention stores the group_id or information that can be used to decipher the group_id in at least the following information: Client-side index: ClientIndex stores the group_id, cache bucket, and position of the client object. In older versions, if this field is not present during index decoding, it defaults to 0.

[0127] Operation logs: The PG operation logs related to write, overwrite, delete, and flush carry the group_id, enabling master and slave replicas to use the same mapping for operations on the same object.

[0128] Missing tracking information: During fault recovery, missing entries are stored in group_id, and the master node uses this to recalculate the group bucket corresponding to the client object.

[0129] Object name reverse lookup: For existing MCache / Backend objects that lack index caching, reverse lookup of group_id from bucket in the object name, i.e., when bucket>B, get group_id=bucket-B.

[0130] Version compatibility: When a new version adds a field, the old version data is decoded as group_id=0; during the upgrade, group requests can be disabled via a function switch to ensure coexistence of the new and old OSDs.

[0131] In summary, this invention proposes a distributed cache data isolation and reclamation method and system based on group identifiers. It transforms the lifecycle rules on the business gateway side into persistent group identifiers, and determines the cache bucket range of client objects in the cache layer and the mapping of large cache objects accordingly. The group identifier is persisted in the index from the first write and is used in subsequent reads, writes, deletions, flushes, pass-throughs, master-slave synchronization, and fault recovery. This allows objects with the same expiration time to be centrally laid out and the space to be released as a whole during batch cleanup, while keeping the original mapping of ungrouped objects unchanged.

[0132] The core technical points of this invention include: Users can create corresponding LC rules in advance based on their business needs, and specify the migration and deletion rules for files / objects on the business gateway side; When a client initiates its first write request for a new small file / object through the business gateway, the gateway finds the matching LC rule based on the object name prefix and generates a corresponding group identifier according to the LC rule; The business gateway sends the group identifier along with the write request to the backend distributed cache layer; The caching layer maps client objects to associated cache objects based on the received group identifier and the storage pool's own information. The caching layer constructs metadata records to establish a mapping relationship between client objects and cached large objects, and persistently saves the group identifier, which cannot be modified during the lifetime of the client object; When the master node processes subsequent read, write, and delete requests for the same object, it does not need to repeatedly carry the group identifier; instead, it automatically obtains it from the object's metadata.

[0133] Furthermore, the present invention also includes: The backend gateway periodically deletes expired objects based on LC rules; since the expiration times are the same, the corresponding group identifiers are also the same, meaning that objects with the same group identifier will be deleted at the same time. After a small file / object is appended to and fills a cache object, the data in the cache object retains the group identifier and is flushed back to the backend large object. The mapping relationship of the small file / object on the client is updated from the cache layer to the backend layer. The mapping algorithm is still calculated based on the group identifier. Multiple cache objects can be merged and flushed back to the same backend object according to the group identifier. When a business gateway initiates a large I / O operation that bypasses the caching layer and directly writes through the cache, the I / O is split into two parts: the aligned part is written directly to the backend pool, and the unaligned part is written to the cache pool. The two parts of the I / O share the same group identifier to ensure the uniqueness of the mapping relationship.

[0134] Compared with the prior art, the significant technical advantages of the present invention include: 1. Support for custom object grouping: Client objects are no longer limited to hash mapping based on object name, but can be mapped to any large cached object through group identifiers, enabling custom object grouping; 2. Store isolation based on business deletion rules: Transform lifecycle rules into transitive and verifiable group identifiers and map them to independent bucket ranges to prevent objects with different deletion cycles from being mixed into the same large cache object; 3. Reduce data gaps and space retention: Collect objects with the same valid expiration time or the same deletion cycle in a group. Large cached objects are more likely to be deleted as a whole after a batch of objects are cleaned up, shortening the space release time. 4. Maintain compatibility with existing mappings: Objects without specified group identifiers continue to use the original hash mapping; group buckets use offset ranges to isolate them from ordinary buckets, reducing the risk of upgrades and migration of existing data; 5. Stable mapping throughout the object's lifecycle: The group identifier written for the first time is written to the index and fixed. Reading, overwriting, deleting, flushing, and restoring all use the saved identifier to avoid position drift of the same object. 6. Covers multiple I / O scenarios: cache append write, cache flush, aligned / unaligned pass-through, and backend object access share the same group identifier to maintain consistency in data location; 7. Enhanced consistency in fault recovery: Preserve or reverse group identifiers in indexes, operation logs, missing data tracking, and object names to reduce the risk of losing group relationships during master-slave synchronization and recovery; 8. Reduce indexing and traversal overhead: The same business rule object is more centrally located in the layout, and lifecycle batch processing, index traversal and cache object cleanup can be organized around group buckets.

[0135] This invention also provides a storage medium for storing a computer program, which, when executed, performs at least the methods described above.

[0136] This invention also provides a control device, including a processor and a storage medium for storing a computer program; wherein the processor executes the computer program by performing at least the method described above.

[0137] This invention also provides a processor that executes a computer program, at least performing the methods described above.

[0138] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc or CD-ROM; magnetic surface memory can be disk storage or magnetic tape storage. The storage media described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0139] In the several embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0140] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0141] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0142] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0143] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0144] The methods disclosed in the several method embodiments provided by this invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0145] The features disclosed in the several product embodiments provided by this invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0146] The features disclosed in the several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0147] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or application, should be considered within the scope of protection of the present invention.

Claims

1. A method for distributed cache data isolation and reclamation based on group identifiers, characterized in that, Includes the following steps: S1. Receive a write request from a client object, and determine the valid expiration condition of the client object by matching the lifecycle rules according to the object name; S2. Generate a group identifier based on the valid expiration conditions. The group identifier is used to identify client objects with the same valid expiration conditions. S3. Pass the group identifier along with the write request to the distributed cache layer; S4. The distributed caching layer calculates the cache key based on the object name of the client object and determines the cache bucket based on the group identifier. The calculation of the cache key is independent of whether a group identifier is specified. When no group identifier is specified, the default mapping algorithm is used to determine the cache bucket. When a group identifier is specified, the group identifier is mapped to a group bucket interval isolated from the default bucket interval. S5. Determine the large cache object based on the cache key and the cache bucket, write the data of the client object into the large cache object, and persist the group identifier and the mapping relationship between the client object and the large cache object to the index; S6. In subsequent requests of the client object, obtain the group identifier from the index, and locate the cached large object based on the group identifier, the cache key, and the cache bucket, so that client objects with the same group identifier are grouped into the same cached large object, and are reclaimed as a whole by group when the lifecycle is deleted.

2. The distributed cache data isolation and reclamation method based on group identifier according to claim 1, characterized in that, In step S1, determining the valid expiration conditions of the client object based on the object name matching lifecycle rules includes: Lifecycle rules are loaded in units of buckets, and the lifecycle rules include object name prefixes and time conditions; Insert the object name prefix character by character into the prefix tree. When matching, match character by character from the root node. When multiple prefix rules are matched at the same time, take the one with the smallest effective deletion period as the matching result. The expiration time of the client object is determined based on the matching results; In step S2, generating a group identifier based on the valid expiration conditions includes: The expiration date of the validity period is normalized by day to obtain a normalized date; The normalized date is encoded as a group identifier, so that objects whose effective expiration time falls on the same day generate the same group identifier.

3. The distributed cache data isolation and reclamation method based on group identifier according to claim 1, characterized in that, In step S4, calculating the cache key and determining the cache bucket includes: The cache key is obtained by performing a modulo operation between the hash value of the object name and the cache key space parameter, and the calculation of the cache key is independent of whether a group identifier is specified. When no group identifier is specified, the default bucket number is obtained by taking the hash value of the object name modulo the number of ordinary buckets; When a group identifier is specified, the number of ordinary buckets is added to the group identifier to obtain the group bucket number, so that the group bucket number falls within an independent interval that does not overlap with the ordinary bucket number interval.

4. The distributed cache data isolation and reclamation method based on group identifier according to claim 1, characterized in that, In step S5, writing the client object's data to the cached large object and persisting the index includes: During the first write operation, the business gateway sends a combined request to the distributed cache layer, consisting of the operation of setting the group identifier and the operation of creating an object or writing data. The distributed caching layer verifies the group identifier, calculates the cache key and cache bucket, and creates or selects the corresponding cache large object. The client object data is written to the cached large object in an append-only manner, and the mapping information of the client object, cached large object, write position and group identifier is persisted in the client index; If the write operation fails, the index is not updated to ensure that the mapping relationship between the group identifier and the object is reliably established.

5. The distributed cache data isolation and reclamation method based on group identifier according to claim 1, characterized in that, In step S6, subsequent requests include read requests and overwrite write requests. Obtaining the group identifier from the index and locating the cached large object includes: When the master node processes a read request or an overwrite request, it first queries the client index; If a group identifier already exists in the index, the group identifier is used directly to recalculate the cache bucket, and the data is read or updated in combination with the cached large object and client object data range recorded in the index. If the group identifier carried in the request is inconsistent with the group identifier stored in the index, the group identifier stored in the index shall prevail.

6. The distributed cache data isolation and reclamation method based on group identifier according to claim 1, characterized in that, In step S6, the overall recycling by group during lifecycle deletion includes: The business gateway generates batch deletion requests for expired objects based on lifecycle rules. Each deletion request locates the corresponding cached large object in the distributed cache layer according to the group identifier stored in the index, and cleans up the mapping relationship between the client object and the cached large object; If there are still mappings to other client objects on the cached large object, only the index relationships of the current object are cleared and the cached large object is retained; if all mapping relationships on the cached large object have been cleared, the cached large object is deleted or its entire space is released. For client objects that have been flushed back to the backend object, clean up the index relationships according to the same rules until the backend object has no remaining mapping relationships and then release the space.

7. The distributed cache data isolation and reclamation method based on group identifier according to claim 1, characterized in that, In step S5, after writing the client object's data to the cached large object, a cache flushing step is also included: When a large cached object reaches a preset capacity or write threshold, a flush is triggered. Create or locate the corresponding backend object based on the cache key, cache bucket, and location range of the original cached large object; Write the data in the cached large object to the backend object and synchronously update the mapping relationship between the client object and the backend object; The group identifier is retained in the flush log or object metadata, or the group identifier is deciphered from the bucket identifier in the backend object name; After a successful refresh, the corresponding cached object in the cache pool is released.

8. The distributed cache data isolation and reclamation method based on group identifier according to claim 1, characterized in that, It also includes the large object pass-through step: For larger I / O requests, split them into aligned and unaligned parts; The aligned portion is written directly to the backend proxy object, and the unaligned portion is written to the cached large object in the cache pool; The aligned portion and the unaligned portion share the same group identifier, cache key, and cache bucket, and the logical range and object name of the two portions are recorded in the metadata. Subsequent reads, overwrites, deletions, and recycling are processed according to the same group identifier.

9. The distributed cache data isolation and reclamation method based on group identifier according to claim 1, characterized in that, In step S5, persisting the group identifier and mapping relationship to the index includes: saving the group identifier, cache bucket, and location of the client object in the client index; The method also includes steps for synchronizing, restoring, and ensuring compatibility of group identifiers: The group identifier is carried in the operation logs related to write, overwrite, delete, and flush, so that the master and slave replicas use the same mapping for operations on the same object; The group identifier is saved in the missing tracking information of fault recovery, and the master node recalculates the group bucket corresponding to the client object accordingly. For objects that already exist but lack an index cache, the group identifier is retrieved from the bucket identifier in the object name; When a version is upgraded or the group bucket resource reaches its limit, new group requests will be downgraded to the default mapping, while existing group buckets will continue to be accessed.

10. A distributed cache data isolation and reclamation system based on group identifiers, characterized in that, This includes a business gateway and a distributed caching layer; The business gateway communicates with the client and the distributed cache layer, and is configured to: receive a write request from a client object, match lifecycle rules according to the object name, determine the valid expiration condition of the client object, generate a group identifier according to the valid expiration condition, and pass the group identifier to the distributed cache layer along with the write request; The distributed caching layer is configured as follows: A cache key is calculated based on the object name of the client object, and a cache bucket is determined based on the group identifier, wherein the calculation of the cache key is independent of whether a group identifier is specified; when no group identifier is specified, a default mapping algorithm is used to determine the cache bucket; when a group identifier is specified, the group identifier is mapped to a group bucket interval isolated from the default bucket interval; a large cache object is determined based on the cache key and the cache bucket, the data of the client object is written into the large cache object, and the group identifier and the mapping relationship between the client object and the large cache object are persisted to the index; in subsequent requests of the client object, the group identifier is obtained from the index, and the large cache object is located based on the group identifier, the cache key, and the cache bucket, so that client objects with the same group identifier are grouped into the same large cache object, and are reclaimed as a whole by group when deleted during their lifecycle.

Images