A method for accessing extremely cold data

Abstract

A method for accessing extremely cold data includes: receiving batch restoration tasks and obtaining metadata about the physical location of objects on the tape; forming an ordered list by globally sorting the tapes and offsets; issuing sequential read commands to the tape drive for consecutive regions based on the list to obtain the raw data stream; performing real-time streaming filtering on the data stream in memory to extract target object data and discard irrelevant data; continuously appending the extracted data to a large aggregate file; and establishing a secondary index for the large aggregate file containing a Bloom filter and an object mapping table to support efficient querying and precise location. This method for accessing extremely cold data achieves sequential large I / O conversion, precise data extraction, and optimized storage structure, effectively solving the performance degradation and hardware wear caused by random access during the restoration of massive amounts of small files on tapes. It significantly improves restoration throughput, reduces hardware wear, and alleviates the pressure on metadata management.

Description

Technical Field

[0001] This invention relates to distributed storage technology, and in particular to a method for accessing extremely cold data. Background Technology

[0002] In the field of distributed storage, data tiering has become a common practice to balance storage costs and access performance. In existing technologies, systems automatically manage data tiers based on access frequency: frequently accessed "hot" data is placed in a high-speed storage pool, infrequently accessed "warm" / "cold" data is placed in a low-speed storage pool, and rarely accessed "extremely cold" data is archived to low-cost media such as tape libraries. When object data is written to tape, its metadata is updated synchronously, recording the object's physical location information on the tape medium. A special marker called FileMark is used on the tape medium to divide logical units in a continuous data stream for rapid location. When accessing archived data, a "restore" operation must be initiated according to standard semantics (such as S3 object storage semantics) and a restore period must be set. Data can be directly read within its validity period; after expiration, it is restored to an archived state. To improve operational efficiency, the system supports batch restore of a group of objects based on object prefixes, tags, and other conditions to avoid the performance and management overhead of individual operations.

[0003] However, in scenarios involving the archiving of massive amounts of small files, the existing "one-to-one object" restoration model has significant drawbacks. Magnetic tape is a typical sequential read / write medium, with seek times often reaching tens of seconds. When restoring a large number of small files discretely stored on magnetic tape, existing solutions require initiating independent location and read requests for each object. This random access mode triggers a severe "shoe-shining" effect. Specifically, the tape drive, unable to maintain a stable forward read / write speed, is forced to frequently accelerate, decelerate, stop, and rewind near the read head, potentially reducing its effective throughput from hundreds of MB / s to a few MB / s, with the majority of time consumed in mechanical positioning. Furthermore, this frequent start-stop and rewinding operations drastically exacerbate the mechanical wear of the magnetic head and tape medium, shortening equipment lifespan and causing noise and temperature rise issues. Simultaneously, the massive number of independent small files generated after restoration can instantly overwhelm the metadata service of the distributed storage system, leading to cluster performance fluctuations.

[0004] In summary, existing technologies face several core challenges when handling the restoration of massive amounts of small files on tape: how to convert inefficient random small I / O (input / output) into tape-friendly sequential large I / O; how to efficiently extract discrete target objects from sequential data streams without reading invalid data; and how to optimize the storage structure of the restored data to reduce metadata management pressure and accelerate subsequent read access.

[0005] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The main objective of this invention is to overcome the deficiencies in the aforementioned background technology and provide a method for accessing extremely cold data.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A method for accessing extremely cold data includes:

[0009] Receive batch restore tasks and obtain metadata of the objects to be restored, the metadata including the physical location information of the objects on the magnetic tape medium;

[0010] The objects to be restored are globally sorted according to the tape they are located on and their physical offset on the tape, forming an ordered read list;

[0011] Based on the ordered read list, a sequential read instruction is sent to the tape drive for a continuous physical area to obtain the raw data stream containing target object data and irrelevant data.

[0012] The raw data stream is filtered in real time in memory to extract the target object data and discard irrelevant data;

[0013] The extracted target object data is continuously appended to a large aggregate file;

[0014] Establishing a secondary index for the aggregated large file includes: constructing a Bloom filter corresponding to the aggregated large file to quickly determine whether an object may exist in the large file; and constructing an object mapping table to accurately record the mapping relationship between the original object identifier and its location information within the aggregated large file.

[0015] A computer program product includes a computer program that, when executed by a processor, implements the method for accessing extremely cold data.

[0016] The present invention has the following beneficial effects:

[0017] This invention proposes a method for accessing extremely cold data, a systematic restoration method involving "global sorting - region scanning - memory filtering - aggregated storage." Its core lies in optimizing tape I / O performance through in-memory computing and reconstructing the data layout to improve access efficiency. This method effectively addresses key pain points in the restoration of massive amounts of small files from tapes: converting inefficient random small I / O into tape-friendly sequential large I / O; efficiently extracting discrete objects from sequential data streams without reading invalid data; and optimizing the structure of the restored data through aggregated storage, reducing metadata pressure on distributed storage and accelerating subsequent readings.

[0018] Compared with traditional technologies, this invention achieves significant technical advantages. First, by merging numerous discrete random read requests into large-span sequential reads across continuous areas of the tape, the "shoe drop effect" is fundamentally eliminated, increasing the recovery throughput from the KB / s level to hundreds of MB / s, the physical limit of the tape, achieving a hundredfold performance leap. Simultaneously, this transformation merges thousands of high-wear start-stop and rewind operations into a single smooth, continuous streaming operation, greatly reducing mechanical wear on the tape drive and media, and significantly extending equipment lifespan. Second, in the data access path, by introducing a resident memory Bloom filter as a first-level index, most invalid queries targeting unrecovered objects can be efficiently intercepted, avoiding direct impact on the backend persistent object mapping table and solving the "read amplification" problem. Furthermore, the storage method of aggregating massive numbers of small files into a few large files significantly reduces the amount of metadata that the distributed file system needs to manage, effectively alleviating the load pressure on metadata services and achieving a good balance between storage efficiency and system performance. Finally, this invention provides a flexible resource management mechanism. Fine-grained expiration management based on object mapping tables allows for the deletion of only the mapping entry when a single object expires, eliminating the need to rewrite the entire aggregate file and decoupling logical expiration from physical garbage collection. Combined with a background lazy garbage collection mechanism, physical storage space is only reclaimed after all associated objects have expired, balancing management efficiency and resource utilization.

[0019] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the overall process architecture of the extremely cold data access method according to an embodiment of the present invention.

[0021] Figure 2 This is a flowchart illustrating the tape region reading strategy and memory streaming filtering process according to an embodiment of the present invention.

[0022] Figure 3 A flowchart illustrating the construction process of the secondary index (Bloom filter + object mapping table) in this embodiment of the invention.

[0023] Figure 4 This is a flowchart illustrating the process of reading and querying aggregated large file objects according to an embodiment of the present invention. Detailed Implementation

[0024] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.

[0025] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0026] This invention aims to address the performance degradation and hardware wear caused by random access when restoring massive amounts of small files on magnetic tape. It proposes a systematic, ultra-cold data access method based on "global sorting - region scanning - memory filtering - aggregated storage." This method transforms discrete, random small I / O operations into large-span sequential reads to eliminate the "shoe drop effect," utilizes memory streaming filtering to accurately extract valid data, and optimizes data layout and access paths through aggregated storage and secondary indexes. This results in a hundredfold increase in throughput, significantly reduced hardware wear, the establishment of an efficient read cache, and support for flexible, fine-grained lifecycle management.

[0027] See Figure 1 This invention provides a method for accessing extremely cold data, comprising the following steps: receiving a batch restoration task; obtaining metadata of the object to be restored, the metadata including the physical location information of the object on the magnetic tape medium; globally sorting the object to be restored according to its location on the tape and its physical offset on the tape to form an ordered read list; based on the ordered read list, initiating a sequential read instruction for a continuous physical area to the tape drive to obtain a raw data stream containing target object data and irrelevant data; performing real-time streaming filtering on the raw data stream in memory to extract target object data and discard irrelevant data; continuously appending the extracted target object data to a large aggregate file; and establishing a secondary index for the large aggregate file, including: constructing a Bloom filter corresponding to the large aggregate file to quickly determine whether an object may exist in the large file; and constructing an object mapping table to accurately record the mapping relationship between the original object identifier and the location information within the large aggregate file.

[0028] In some embodiments, the global sorting of the objects to be restored includes: grouping all objects to be restored into different processing queues according to the tape serial number in the metadata; sorting the objects in ascending order according to the logical block offset on the tape within the processing queue corresponding to each tape; and logically merging the reading ranges of multiple objects that are physically adjacent in the sorted list and within a preset continuous reading threshold to form an optimized continuous reading interval.

[0029] In some embodiments, initiating a sequential read instruction based on an ordered read list includes (see...) Figure 2 The upper part of the process divides data segments based on file markers written at capacity intervals on the magnetic tape medium; it determines whether adjacent objects to be read in the ordered list are located within the same file marker interval; if they are located within the same interval, it initiates a continuous streaming read operation on that interval; if it crosses one or more file marker intervals and the interval exceeds a set threshold, it sends a fast positioning command to the tape drive to skip the intermediate file markers and invalid data areas, directly locate the beginning of the next file marker interval containing the target object, and then performs continuous reading.

[0030] In some embodiments, the real-time streaming filtering in memory includes (see...) Figure 2 (Lower half): The original data stream is written into a memory circular buffer; the data flowing through the buffer is compared in real time according to the table of precise location intervals of the objects to be restored, which is prefetched from metadata and loaded into memory; when the logical offset of the data falls into the location interval of a target object, the data is copied to a buffer dedicated to aggregation; when the logical offset of the data is outside the target object interval, the data is discarded directly; the valid object data accumulated in the aggregation buffer is continuously stored in the aggregation large file by appending.

[0031] In some embodiments, see Figure 3 The construction of the secondary index includes: creating an independent and resident memory Bloom filter instance for each aggregated large file, and inserting only the hash value of the original object identifier contained therein into this filter when the large file is generated; in the distributed key-value storage system, maintaining an object mapping table for each aggregated large file, wherein the key is the original object identifier, and the value is a structure that at least contains the identifier of the aggregated large file, the starting offset of the object in the large file, and the length of the object data.

[0032] In some embodiments, see Figure 4The extremely cold data access method further includes an object reading process: receiving a read request for the original object; concurrently traversing all active Bloom filter instances and querying the original object identifier; if all Bloom filter instances return negative results, it is determined that the object is not stored in any aggregated large file, and the read process is executed based on its original metadata; if any Bloom filter instance returns a positive result, the object mapping table associated with that instance is queried sequentially; if an exact mapping entry is found in the object mapping table, the object data is read from the corresponding aggregated large file and returned based on the position information in the entry; if no mapping entry is found in the object mapping table, it is determined to be a false positive by the Bloom filter, and the object mapping tables associated with other Bloom filters that return positive results are checked; if all associated object mapping tables are not matched, the process reverts to the read process executed based on the original metadata (if the object is still in an archived state, a restore operation is initiated first). This hierarchical query method, which combines fast Bloom filter filtering, accurate Omap verification, and original metadata fallback, effectively minimizes the average read latency.

[0033] In some embodiments, the extremely cold data access method further includes a lifecycle management step: when the restore validity period of a single original object expires, only logical deletion is performed, that is, the mapping entry of the object is removed from the corresponding object mapping table, without immediately operating on the aggregated large file or updating the Bloom filter; the status of each object mapping table is periodically scanned through a background garbage collection process; only when all entries in the object mapping table corresponding to a certain aggregated large file have been removed, that is, when the large file is no longer associated with any valid object, is the physical deletion operation of the aggregated large file and its corresponding Bloom filter instance triggered.

[0034] In some embodiments, the aggregated large file only sequentially stores the data entities of the target objects and does not contain any built-in metadata describing the object boundaries; the size and boundary information of the objects are completely described by the external object mapping table.

[0035] In some embodiments, the acquisition of metadata of the object to be restored is a batch prefetch operation, and the global sorting is performed in memory in preparation for the generation of the sequential read instruction.

[0036] This invention proposes an ultra-cold data access method that effectively solves the core challenge of restoring massive amounts of small files on magnetic tape. Through global sorting and continuous region reading, it transforms discrete, random small I / O into tape-friendly sequential large I / O, fundamentally eliminating the "shoe drop effect" and increasing restoration throughput from the KB / s level to the physical limits of magnetic tape, achieving a hundredfold performance leap and significantly reducing mechanical wear on tape drives. It utilizes memory streaming filtering to accurately extract target data, and combines a resident memory Bloom filter with a secondary index composed of a persistent object mapping table (Omap) to establish an efficient read path, effectively intercepting invalid queries and avoiding read amplification impact. Simultaneously, by aggregating massive amounts of small files into large files, it significantly reduces the metadata management pressure on distributed storage systems and, relying on Omap, achieves fine-grained logical expiration and lazy physical reclamation without rewriting large files, thus achieving an excellent balance between storage efficiency, access performance, and resource management flexibility.

[0037] The following further describes the implementation methods and advantages of specific embodiments of the present invention.

[0038] The extremely cold data access method of this invention is a systematic restoration method consisting of "global sorting - region scanning - memory filtering - aggregated storage", and its overall process architecture is as follows: Figure 1 As shown. Its core idea is to optimize disk (tape) I / O performance through in-memory computing and restructure the data layout to improve access efficiency.

[0039] According to some embodiments of the present invention, a distributed storage data restoration method based on global sorting and streaming aggregation is provided. The method first receives batch restoration tasks and obtains the metadata of the objects to be restored; it groups the objects according to their physical tape numbers in the metadata, and then sorts them in ascending order within each group according to their physical offsets on the tape, completing global scheduling. Subsequently, it calculates the minimum start offset and maximum end offset of the sorted object list, and accordingly sends a single read command to the tape drive covering the entire contiguous region.

[0040] The acquired data stream is parsed and filtered in real time in a memory buffer. Based on a pre-defined object location information table, the system extracts target data fragments and discards irrelevant gaps. All extracted discrete object data is continuously appended to the same aggregated large file, achieving physical aggregation and encapsulation of the data. To support efficient retrieval of this aggregated large file, a secondary index needs to be constructed: firstly, a mapping table is established from the original object identifier to its precise offset and size within the aggregated large file; secondly, an auxiliary, resident Bloom filter is constructed. When processing data read requests, the Bloom filter is queried first to quickly determine if the object exists. Only when the filter indicates a possible existence is the mapping table in persistent storage further queried to locate and read the data.

[0041] Furthermore, the magnetic tape medium is provided with file markers distributed at fixed capacity intervals. When performing continuous area reading, if adjacent objects to be restored are located within the same file marker interval, continuous streaming reading is performed; if they span one or more file marker intervals, the system quickly skips these markers and the invalid data areas in between through specific instructions, and directly locates the start of the next interval containing the target object, thereby avoiding the transmission of invalid data.

[0042] In terms of data management, when the restoration lifetime of a single original object expires, its corresponding entry is only logically deleted from the mapping table. The system only performs a physical deletion operation on a large aggregate file, completing space reclamation, after all mapping entries associated with that large aggregate file have been deleted. Furthermore, the large aggregate file employs a compact data arrangement format, storing only the object data entities themselves sequentially, without containing any built-in metadata describing object boundaries; the size and boundary information of the objects are entirely recorded and described by an external mapping table.

[0043] The following section, in conjunction with specific technical modules, further elaborates on the internal working mechanism and implementation details of the key steps in the above method.

[0044] Task scheduling and global sorting:

[0045] Users submit batch restore tasks with specific filtering criteria. Upon receiving the batch restore tasks, the storage system first enters a buffer scheduling phase. During this phase, it retrieves the physical location metadata of all objects to be restored from the metadata engine in batches, including tape serial numbers, block offsets, and data lengths. A two-level sorting process is then performed: first, tasks are distributed to the corresponding processing queues based on tape serial numbers; then, within each queue, they are sorted in strict ascending order according to block offsets. Furthermore, the system performs merge optimization, logically treating multiple objects located between two adjacent FileMarks as a single contiguous data block and merging their read ranges to avoid minor tape start / stop operations.

[0046] Region-Aware Sequential Read:

[0047] For single-disc tapes, the scheduler leverages the physical characteristic of writing a FileMark approximately every 2GB during archiving, employing a "segment aggregation, inter-segment skipping" strategy. Its execution logic is as follows: Figure 2As shown in the diagram. The core of this strategy is to calculate the continuous tape segments defined by the start position of the first object and the end position of the last object based on the sorted list of objects, and then initiate sequential read commands accordingly. Within a FileMark segment, for multiple objects falling between two adjacent FileMarks, regardless of the actual interval size, the system tends to merge them into a single continuous read command. This is because within a smaller data range, the overhead of continuous reading and subsequent filtering is lower than performing multiple start-stop operations. When the next object to be read crosses a FileMark and is significantly distant from the current position, the system uses the SCSI SPACE command to quickly skip the intermediate FileMarks and invalid data areas, directly locating the start position of the next target FileMark, rather than sequentially reading invalid data. This execution logic allows the tape drive to maintain a streaming read mode within a segment and achieve rapid location when crossing segments, thus achieving an efficient access effect of "macroscopic jumps and microscopic continuity."

[0048] In-Memory Stream Filtering:

[0049] The raw data stream read from the tape contains target object data and irrelevant gap data. This data stream is fed into a memory circular buffer, and the filtering engine performs sliding window filtering based on a "table of object intervals to be restored," such as... Figure 2 The lower half is shown. When the offset of the data flowing through the buffer falls within the start and end range of a target object, that portion of the data is copied to a dedicated aggregation buffer; if the data falls into an irrelevant range, it is discarded directly. All extracted valid object data is continuously appended to a new "large file," completing the data reorganization and encapsulation.

[0050] Two-Level Indexing Acceleration and Metadata Mapping:

[0051] To facilitate rapid location of the original object from the aggregated large file, this invention constructs a two-level index, the construction and mapping relationship of which is as follows: Figure 3As shown. The first-level index L1 is a Bloom filter, with each large file having its own independent Bloom filter instance. When a large file is created, only the object identifiers contained within that file are inserted into their corresponding filters. When processing read requests, the system concurrently traverses all active Bloom filters in memory; if any filter returns "may exist," a query to the second-level index is triggered. In this architecture, the Bloom filter acts as the first line of defense for read requests, quickly filtering out a large number of queries for objects that definitely do not exist with very low overhead. When a large file is deleted, its corresponding filter instance is discarded directly. The second-level index L2 is an object mapping table, maintained for each large file in the distributed key-value storage component. Its key is the original object identifier, and the value is a structure containing the large file ID, the object's offset within the large file, and the object's size. This mapping table provides precise location information and is only queried for requests that have passed the initial screening by the Bloom filter.

[0052] Object reading process:

[0053] When a user initiates an object read request, the system performs a hierarchical query. The process is as follows: Figure 4 As shown. First, all active Bloom filters are concurrently traversed. If all filters return negative results, it is quickly determined that the object has not been restored, and the system will fall back to the reading process based on the original metadata (if the object is in archive state, restoration must be initiated first). If any filter returns a positive result (which may exist), its associated object mapping table (Omap) is further queried. If an exact mapping entry is found in the Omap, the data is read from the corresponding aggregate file and returned based on its location information. If it is not found in the Omap (i.e., a false positive from the Bloom filter), the Omap associated with other filters that returned positive results is checked; if all relevant Omaps are not found, the system finally falls back to the original metadata reading process. This process achieves fast path through Bloom filters, verifies and confirms through Omap, and has a complete fallback mechanism, thereby ensuring data accessibility while minimizing average read latency.

[0054] Lifecycle and space recycling:

[0055] The system employs a two-tiered mechanism to manage data lifecycle. When the time-to-live (TTL) of a single object expires, only a logical expiration operation is performed, i.e., the key-value entry for that object is deleted from the corresponding object mapping table. This is a lightweight key-value deletion operation that does not immediately affect the physical data. This design is for performance reasons, as deleting elements from a Bloom filter is costly; since subsequent Omap queries can ultimately confirm the existence of the object, omitting the filter update is acceptable. Physical space reclamation is achieved through a lazy garbage collection mechanism. The background garbage collection process periodically scans the status of each object mapping table. Only when the object mapping table entry corresponding to a large aggregate file is empty, or the proportion of valid data within it is lower than a certain threshold (which may trigger a compression and rewrite operation), will the physical deletion or data reorganization of that large aggregate file and its associated Bloom filter instances be triggered, thereby achieving the final reclamation of storage space.

[0056] Example

[0057] The workflow of this invention is further illustrated below with an example. Suppose a user needs to restore 10,000 image files (img_001.jpg to img_10000.jpg), each 200KB in size. These files are stored scattered on a magnetic tape with serial number TAPE_A, with logical locations roughly ranging from 10GB to 1TB.

[0058] During the scheduling phase, the system queries metadata to obtain the location of all objects, then sorts them by tape offset to generate an ordered read list: [ {img_001, 10GB}, {img_052, 10.2GB}, ..., {img_10000, 1TB}]. During the reading phase, the system calculates the overall continuous read range to be 10GB to 1TB (a span of 990GB) and sends a one-time continuous read command to the tape drive. After the drive is located, data acquisition is completed in about 50 minutes at a speed of approximately 300MB / s. In contrast, if the traditional object-by-object reading method were used, it would be estimated that tens of thousands of start-stop positioning operations would be required, with a total time exceeding 60 hours.

[0059] During the processing phase, the system performs real-time filtering on the 990GB raw data stream read in memory. Once the filtering engine identifies target object data (such as the 200KB data in img_001), it appends it to a new aggregated large file named BigFile_X, while discarding non-target data (such as a 300MB gap in the data). Ultimately, all 10,000 objects are aggregated into a single approximately 2GB BigFile_X file.

[0060] During the indexing phase, the system constructs a separate Bloom filter, Bloom_X, for BigFile_X and inserts the hash values ​​of all 10,000 object identifiers into it. Simultaneously, precise mappings are established in the object mapping table; for example, img_001 maps to [{BigFile_X, Offset=0, Size=200KB}], img_052 maps to [{BigFile_X, Offset=4MB, Size=200KB}], and so on.

[0061] During the access phase, when a user requests to read img_052, the process is as follows: Figure 4 As shown, the system first iterates through the Bloom filter, and Bloom_X returns "possibly exists" (True). Next, it queries the corresponding object map to obtain the precise offset of 4MB. Finally, the system directly reads and returns the object data from the 4MB offset in the BigFile_X file. If the object is not restored or has expired, the Bloom filter or object map will give a negative result, and the system will revert to the original tape read process.

[0062] In summary, this invention offers significant technical advantages through its systematic approach of "global sorting - region scanning - memory filtering - aggregated storage." First, in small file archiving and restoration scenarios, by converting numerous discrete random I / O requests into tape-friendly, large-span sequential reads, the "shoe drop effect" is effectively eliminated, thereby increasing the actual restoration throughput from the KB / s level to the physical limits of tape—hundreds of MB / s (300-400 MB / s)—achieving a hundredfold performance leap. Second, this method merges thousands of high-wear start-stop and rewind operations into a single smooth, continuous streaming operation, greatly reducing the mechanical wear of the tape drive and media, and significantly extending the hardware's mean time between failures (MTBF). On the read path, by introducing a resident memory Bloom filter as a primary index, the vast majority (99%) of invalid queries to unrestored objects can be efficiently intercepted, avoiding the "read amplification" problem caused by directly impacting the backend persistent object mapping table. Furthermore, the storage method of aggregating massive numbers of small files into a few large files (e.g., one large file vs. 10,000 small files) significantly reduces the amount of metadata that the distributed file system needs to manage, effectively alleviating the load pressure on the metadata service (MDS) and achieving a good balance between storage space utilization and access performance. Finally, the fine-grained expiration management mechanism based on the object mapping table (Omap) ensures that the failure of a single object only requires deleting the mapping entry, without rewriting the entire aggregated large file, thus achieving flexible and efficient resource reclamation and lifecycle management.

[0063] This invention also provides a storage medium for storing a computer program, which, when executed, performs at least the methods described above.

[0064] This invention also provides a control device, including a processor and a storage medium for storing a computer program; wherein the processor executes the computer program by performing at least the method described above.

[0065] This invention also provides a processor that executes a computer program, at least performing the methods described above.

[0066] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc or CD-ROM; magnetic surface memory can be disk storage or magnetic tape storage. The storage media described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0067] In the several embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0068] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0069] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0070] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0071] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0072] The methods disclosed in the several method embodiments provided by this invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0073] The features disclosed in the several product embodiments provided by this invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0074] The features disclosed in the several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0075] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or application, should be considered within the scope of protection of the present invention.

Claims

1. A method for accessing extremely cold data, characterized in that, include: Receive batch restore tasks and obtain metadata of the objects to be restored, the metadata including the physical location information of the objects on the magnetic tape medium; The objects to be restored are globally sorted according to the tape they are located on and their physical offset on the tape to form an ordered reading list; specifically, this includes: grouping all objects to be restored into different processing queues according to the tape serial number in the metadata; and sorting the objects in ascending order according to the logical block offset on the tape within the processing queue corresponding to each tape. Based on the ordered read list, a sequential read instruction is sent to the tape drive for a continuous physical area to obtain the raw data stream containing the target object data and irrelevant data. Specifically, this includes: dividing the data segment according to the file markers written on the tape medium at capacity intervals; determining whether adjacent objects to be read in the ordered list are located within the same file marker interval; if they are located within the same interval, a continuous streaming read operation is initiated for that interval; if one or more file marker intervals are crossed and the interval exceeds a set threshold, a fast positioning instruction is sent to the tape drive to skip the intermediate file markers and invalid data areas, directly locate the beginning of the next file marker interval containing the target object, and then perform continuous reading. The raw data stream is filtered in real time in memory to extract the target object data and discard irrelevant data; The extracted target object data is continuously appended to a large aggregate file; Establishing a secondary index for the aggregated large file includes: constructing a Bloom filter corresponding to the aggregated large file to quickly determine whether an object may exist in the large file; and constructing an object mapping table to accurately record the mapping relationship between the original object identifier and its location information within the aggregated large file.

2. The method as described in claim 1, characterized in that, The global sorting of the objects to be restored also includes: For multiple objects that are physically adjacent in the sorted list and fall within a preset continuous reading threshold, their reading ranges are logically merged to form an optimized continuous reading interval.

3. The method as described in claim 1, characterized in that, The step of performing real-time streaming filtering on the original data stream in memory to extract target object data and discard irrelevant data specifically includes: The original data stream is written into a memory circular buffer; Based on the table of precise location ranges of the objects to be restored, which are prefetched from metadata and loaded into memory, the data flowing through the buffer is compared in real time. When the logical offset of the data falls within the location range of a target object, this part of the data is copied to a buffer dedicated to aggregation; If the logical offset of the data is outside the target object range, then that part of the data is discarded directly. The accumulated valid object data in the aggregation buffer is continuously stored into the aggregated large file in an append-only manner.

4. The method as described in claim 1, characterized in that, The step of establishing a secondary index for the aggregated large file specifically includes: Create a separate, resident Bloom filter instance for each large aggregate file, and insert only the hash value of the original object identifier contained in the file into this filter when the large file is generated; In a distributed key-value storage system, an object mapping table is maintained for each aggregated large file. The key is the original object identifier, and the value is a structure that contains at least the identifier of the aggregated large file, the starting offset of the object within the large file, and the length of the object data.

5. The method as described in claim 1, characterized in that, It also includes the object reading process: Receive read requests for the original object; Concurrently traverse all active Bloom filter instances and query the original object identifier; If all Bloom filter instances return negative results, it is determined that the object is not stored in any aggregate file, and the reading process is instead performed based on its original metadata; If any Bloom filter instance returns a positive result, the object mapping table associated with that instance is queried sequentially. If a precise mapping entry is found in the object mapping table, the object data is read from the corresponding aggregated large file and returned based on the position information in the entry; If no mapping entry is found in the object mapping table, it is determined to be a false alarm of the Bloom filter, and the object mapping tables associated with other Bloom filters that return positive results are checked. If all associated object mapping tables are not matched, the process falls back to the read flow performed based on the original metadata.

6. The method as described in claim 1, characterized in that, It also includes lifecycle management steps: When the restore validity period of a single original object expires, only logical deletion is performed, that is, the mapping entry of the object is removed from the corresponding object mapping table, without immediately operating on the aggregated large file or updating the Bloom filter; The background garbage collection process periodically scans the status of each object mapping table. The physical deletion operation of a large aggregate file and its corresponding Bloom filter instance is triggered only when all entries in the object mapping table corresponding to a large aggregate file have been removed, that is, when the large file is no longer associated with any valid object.

7. The method as described in claim 1, characterized in that, The aggregated large file only stores the data entities of the target objects sequentially and does not contain any built-in metadata describing the object boundaries; the size and boundary information of the objects are completely described by the external object mapping table.

8. The method as described in claim 1, characterized in that, The acquisition of metadata of the object to be restored is a batch prefetch operation, and the global sorting is completed in memory in preparation for the generation of the sequential read instruction.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the method for accessing extremely cold data as described in any one of claims 1 to 8.

Images