A directory de-hierarchy method and system for a hierarchical storage system

By using full preheating and file-by-file state transition, the problems of data silos and task conflicts after replacing the S3 storage system were solved, ensuring data security and business continuity, and improving operation and maintenance efficiency.

CN123067801APending Publication Date: 2026-11-10BEIJING YAN RONG TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611592982.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-15
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

When replacing the S3 storage system backend, existing technologies have problems such as data silos, non-atomic switching of hierarchical association attributes, task conflicts during data migration, complexity of recursive directory processing, and lack of flexible file-by-file processing granularity, which can lead to data inaccessibility and business interruption.

Method used

It adopts a method of full preheating and file-by-file atomic state transition. By generating a list of directories to be processed, it detects and intercepts task conflicts, traverses the directory tree to preheat and update the file status, and ensures that all data exists completely in local storage. It supports multi-threaded concurrent preheating to accelerate data migration.

Benefits of technology

It eliminates the risk of data inaccessibility, ensures business continuity and task consistency, ensures the complete removal of the hierarchical attributes of the directory tree, and improves operational efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123067801A_ABST
    Figure CN123067801A_ABST
Patent Text Reader

Abstract

This specification discloses a directory de-stratification method and system for a hierarchical storage system, relating to the field of storage technology. The method includes: obtaining a directory de-stratification request; performing preprocessing based on the directory de-stratification request to generate a list of directory trees to be processed; performing task conflict detection and interception on the list of directory trees to be processed, and setting interception flags in the list of directory trees to be processed; traversing the list of directory trees to be processed, performing preheating to generate a preheating list, performing state transitions on the files in the preheating list, updating the file states in the preheating list and clearing the hierarchical flags; determining whether the list of directory trees to be processed has been completely de-stratified, and if the list of directory trees to be processed has been completely de-stratified, clearing the interception flags set in the list of directory trees to be processed and restoring task scheduling.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This specification relates to the field of storage technology, and in particular to a directory de-layering method and system for a hierarchical storage system. Background Technology

[0002] Distributed file storage systems typically employ a hierarchical storage management (HSM) architecture to meet the capacity and cost requirements of massive data storage. In this architecture, frequently accessed hot data is stored on high-performance local storage media, while infrequently accessed cold data is migrated to low-cost external object storage, thus achieving an optimal balance between storage cost and access performance. Users access data through file system interfaces (POSIX, NFS, SMB, etc.). The system automatically or manually triggers the flow of data between local storage and object storage based on the data's accessibility, performing loading, flushing, and warm-up operations.

[0003] To achieve the aforementioned hierarchical management capabilities, the system adds a hierarchical identifier to the metadata records of the distributed file system to mark which S3 bucket a directory or file is associated with. Once a directory is marked as a hierarchical directory, the hierarchical behavior of its files is controlled by directory attribute inheritance. However, in real-world applications, there may be situations where it's necessary to replace the existing S3 storage system. Replacing the S3 storage backend will present a series of challenges, including the risk of data silos, non-atomic switching of hierarchical association attributes, task conflicts during data migration, the complexity of recursive directory processing, and a lack of flexible file-by-file processing granularity.

[0004] Therefore, a directory de-layering method and system for a hierarchical storage system are needed. Summary of the Invention

[0005] This specification provides a directory de-layering method and system for a hierarchical storage system to solve the above-mentioned technical problems.

[0006] To solve the above-mentioned technical problems, the embodiments in this specification are implemented as follows: This specification provides an embodiment of a directory de-layering method for a hierarchical storage system, including: Get directory de-layering request; Based on the directory de-hierarchy request, preprocessing is performed to generate a list of directory trees to be processed; Perform task conflict detection and interception on the list of directories to be processed, and set interception flags in the list of directories to be processed; Traverse the list of directories to be processed, perform preheating to generate a preheating list, perform state transitions on the files in the preheating list, update the file states in the preheating list, and clear the hierarchical markers; Determine whether the list of directory trees to be processed has been completely de-layered. If the list of directory trees to be processed has been completely de-layered, then clear the interception flags set in the list of directory trees to be processed and restore task scheduling. If the list of pending directory trees has not been completely de-layered, a new directory de-layering request will be initiated for the list of failed files until the list of pending directory trees has been completely de-layered.

[0007] This specification also provides a directory de-hierarchy system for a hierarchical storage system, comprising: Get the module, get the directory de-layering request; The preprocessing module performs preprocessing based on the directory de-hierarchy request to generate a list of directory trees to be processed; The interception module performs task conflict detection and interception on the list of directories to be processed, and sets an interception flag in the list of directories to be processed. The state transition module traverses the list of directories to be processed, performs preheating to generate a preheating list, performs state transitions on the files in the preheating list, updates the file status in the preheating list, and clears the hierarchical markers. The directory attribute processing module determines whether the list of directory trees to be processed has been completely de-layered. If the list of directory trees to be processed has been completely de-layered, the interception flags set in the list of directory trees to be processed are cleared and task scheduling is restored. If the list of pending directory trees has not been completely de-layered, the re-request module will re-initiate a directory de-layering request for the failed file list until the list of pending directory trees has been completely de-layered.

[0008] The directory de-stratification method of the tiered storage system provided in this specification ensures that all data exists completely in local storage before the directory is restored to a normal directory by forcibly performing a full warm-up before removing the tiered attributes. This eliminates the risk of data inaccessibility due to disconnection of the S3 interface, thereby improving data security. During the warm-up period, users access incomplete files and fetch them in real time as needed. Newly written files are directly stored in the Local state. The de-stratification operation is imperceptible to normal business read and write operations, ensuring business continuity during storage backend migration. By automatically intercepting flush / export tasks, data corruption or inconsistency caused by concurrent tasks changing file states during the warm-up process is avoided, thus ensuring task consistency. All nested subdirectories are automatically processed to ensure that the tiered attributes of the entire directory tree are completely removed, eliminating any lingering issues of inconsistent tiered attributes between parent and child directories. Configurable multi-threaded / multi-process concurrent warm-up is supported, allowing users to dynamically adjust the concurrency based on network bandwidth and system resources, accelerating large-scale data migration. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This specification provides a schematic diagram of the system architecture for a directory de-layering method of a layered storage system, as illustrated in an embodiment of the present specification. Figure 2 A flowchart illustrating a directory de-layering method for a hierarchical storage system provided in an embodiment of this specification; Figure 3 This is a schematic diagram of a directory de-hierarchy system provided in an embodiment of this specification. Detailed Implementation

[0011] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0012] In real-world applications of distributed file storage systems, various factors can necessitate replacing the existing remote storage system (e.g., S3 storage) due to: storage provider switching (e.g., migrating from AWS S3 to Alibaba Cloud OSS); storage cost optimization (switching to a lower-cost storage service); data center migration or disaster recovery architecture adjustments; or storage performance failing to meet business needs, requiring an upgrade to higher-performance object storage. When users need to disconnect from the old S3 system, directly deleting the directory's hierarchical metadata or breaking the S3 association will cause files already in S3 status to lose their data access path. The actual data of these files will only exist in the old S3 storage, while only metadata or stubs will be available locally. Once the old S3 system goes offline or access credentials expire, these files will become permanently inaccessible, leading to severe data loss and the risk of data silos.

[0013] Existing systems often remove the "hierarchical association attribute" from directories by directly modifying metadata tags. While this operation is atomic, it doesn't account for the differences in file states within a directory—a directory may simultaneously contain Local, S3, and mixed-state files. Directly marking a directory as a "non-hierarchical directory" will lead to: loss of hierarchical information for S3-state files, making it impossible to trace the files' remote storage location; and discrepancies between file states and directory attributes, resulting in inconsistencies in the system's internal state and thus causing non-atomic switching of the hierarchical association attribute.

[0014] If the system simultaneously executes a flush task in the same directory while pulling S3 data to the local machine—that is, flushing the local data back to S3—a data consistency problem will occur: the file may be in a contradictory state of "pulling from S3" and "pushing to S3". This may result in data version inconsistencies or even data corruption, ultimately leading to task conflicts during data migration.

[0015] Processing recursive directories is complex, as hierarchical directories may contain nested subdirectories, with subdirectories inheriting the hierarchical attributes of their parent directories. When de-hierarchizing, if only the top-level directory is processed without recursively processing all subdirectories, hierarchical relationships will remain in the subtree, resulting in an abnormal state of "inconsistent hierarchical attributes between parent and child directories."

[0016] Existing solutions lack flexible file-by-file processing granularity. They typically remove hierarchical attributes on a directory-by-directory basis. If any file in the directory fails to be converted, the entire operation fails and requires a complete rollback, causing users to have to try repeatedly, resulting in low operational efficiency.

[0017] Based on this, this specification provides a method for de-stratifying a directory in a hierarchical storage system. When replacing the object storage backend, the method achieves safe removal of directory hierarchical attributes through full preheating and file-by-file atomic state transition.

[0018] Figure 1 This is a schematic diagram of the system architecture for a directory de-layering method of a layered storage system provided in an embodiment of this specification. For example... Figure 1 As shown, system architecture 100 may include terminal device 101, terminal device 102, terminal device 103, network 104, and server 105. Network 104 is used as a medium to provide communication links between terminal device 101, terminal device 102, terminal device 103, and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0019] Terminal devices 101, 102, and 103 interact with server 105 via network 104 to receive or send messages, etc. Various client applications can be installed on terminal devices 101, 102, and 103, such as dedicated programs for directory de-layering methods based on hierarchical storage systems.

[0020] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various dedicated or general-purpose electronic devices, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. They can be implemented as multiple software programs or software modules (e.g., multiple software programs or software modules used to provide distributed services) or as a single software program or software module.

[0021] Server 105 can be a server that provides various services, such as a backend server that provides services to client applications installed on terminal devices 101, 102, and 103. For example, the server can de-layer the directory of a hierarchical storage system so that the de-layering results can be displayed on terminal devices 101, 102, and 103.

[0022] Server 105 can be either hardware or software. When server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 105 is software, it can be implemented as multiple software programs or software modules (e.g., multiple software programs or software modules used to provide distributed services), or as a single software program or software module.

[0023] Figure 2This is a flowchart illustrating a directory de-layering method for a hierarchical storage system provided in an embodiment of this specification. From a programming perspective, the execution entity of the process can be a program hosted on an application server or application terminal. It is understood that this method can be executed by any device, equipment, platform, or cluster of devices with computing and processing capabilities. Figure 2 As shown, the method for de-hierarchizing the directory includes: Step S201: Obtain the directory de-layering request.

[0024] In this illustrated embodiment, the directory de-stratification request is initiated by the user to the distributed tiered storage system, with the purpose of removing the tiered association between the target file and remote storage (such as the S3 storage system).

[0025] In specific embodiments, users can initiate directory de-layering requests through command lines, web consoles, or API interfaces. The specific method of initiating directory de-layering requests is not limited here and does not constitute a specific limitation on this application.

[0026] In the embodiments of this specification, the directory de-layering request includes: the path of the target layered directory, a recursion flag, and the number of concurrent warm-up threads. The path of the target layered directory is an absolute path; recursion is generally set to true by default to ensure that all nested subdirectories are processed; the number of concurrent warm-up threads refers to the number of threads executing warm-up tasks in parallel during the warm-up phase. In specific embodiments, the number of concurrent warm-up threads is configured by the user based on network bandwidth, and is generally set to 4 or 8 by default.

[0027] In one embodiment, in a distributed file storage system, users can initiate directory de-hierarchy requests via command-line tools (CLI), a web console, or an API interface. Request parameters include at least: target_path: the absolute path to the target directory (e.g., / data / project_a); recursive: a recursive flag, which is defaulted to and forced to true in this invention to ensure all nested subdirectories are processed; concurrency: the number of concurrent warm-up threads, which users can configure based on network bandwidth (default values ​​can be 4 or 8).

[0028] After receiving the request, the system first verifies whether the target directory exists and has hierarchical association attributes (i.e., the is_tiered flag in the directory metadata is true). If the directory is not a hierarchical directory, the system directly returns an error, prompting the user that there is no need to perform the de-hierarchical operation.

[0029] Step S203: Based on the directory de-hierarchy request, perform preprocessing to generate a list of directory trees to be processed.

[0030] After obtaining the directory de-stratification request, the first step is to verify whether the target directory exists and has a hierarchical association attribute. If the target directory exists and has a hierarchical association attribute, the subsequent de-stratification steps are executed; if the target directory does not exist or does not have a hierarchical association attribute, an error is returned and the subsequent de-stratification operation is not executed.

[0031] In this embodiment of the specification, the preprocessing based on the directory de-hierarchy request to generate a list of directory trees to be processed specifically includes: Based on the directory de-layering request, the target hierarchical directory with hierarchical association attributes is determined; The target hierarchical directory and its nested subdirectories are traversed to generate a list of directory trees to be processed.

[0032] In this embodiment of the specification, the step of traversing the target hierarchical directory and its nested subdirectories to generate a list of directory trees to be processed specifically includes: Using the target hierarchical directory as the root node, the nested subdirectories of the target hierarchical directory are traversed to generate the list of directories to be processed. The metadata of the list of directories to be processed includes: directory path, hierarchical identifier, associated remote storage name, remote storage path prefix, total number of files, and number of files initially in remote / mixed state.

[0033] The target hierarchical directory and its nested subdirectories are traversed to generate a list of directory trees to be processed. Specifically, either depth-first or breadth-first search can be used to traverse the target hierarchical directory and its nested subdirectories.

[0034] Regardless of whether a depth-first or breadth-first search approach is used, the target hierarchical directory is the root node, and its nested subdirectories are the child directory nodes. Traversing all root nodes and child directory nodes, as well as nested subdirectories, ensures that the hierarchical attribute of the entire directory tree is completely removed. During the traversal of each child directory node, the corresponding metadata information is recorded, ultimately generating a list of directory trees to be processed. Specifically, the metadata information for each child directory node includes: hierarchical depth, total number of files, and number of S3 / MIX status files, for subsequent progress display and logging.

[0035] In one embodiment, the system uses the target directory as the root node and performs a depth-first or breadth-first traversal to generate a complete list of directories to be processed. ,in For the target directory, This includes all its nested subdirectories. The metadata for each directory to be processed contains: dir_path: directory path; is_tiered: tier identifier (boolean); tiering_bucket: associated S3 bucket name; tiering_prefix: remote storage path prefix; file_count: total number of files in the directory; tiered_file_count: number of files in S3 / mixed state.

[0036] Step S205: Perform task conflict detection and interception on the list of directories to be processed, and set an interception flag in the list of directories to be processed.

[0037] In this embodiment of the specification, the step of performing task conflict detection and interception on the list of directories to be processed, and setting an interception flag in the list of directories to be processed, specifically includes: Query the operation tasks under the pending directory tree list, the operation tasks include brushing tasks and export tasks; Traverse the active tasks in the operation tasks and mark the tasks that are in the PENDING and / or RUNNING states as ABORTED. Set an interception flag in the metadata of the root directory of the pending directory tree list, and synchronize the interception flag to all subdirectories in the pending directory tree list.

[0038] The purpose of task conflict detection and interception is to automatically stop and intercept the brushing / exporting tasks, so as to avoid abnormal data status caused by task conflicts during the subsequent warm-up process.

[0039] In the specific implementation process, the operational entities involved in task interception include: users, de-stratification controllers, task scheduling modules, directory metadata services, and execution threads. After a user initiates a de-stratification request, the controller queries the task scheduling module for flush tasks under the pending directory tree. Flush tasks refer to migrating cold data from the file system to the associated object storage (S3) and releasing space in local storage. Export tasks refer to exporting file data from a single target to external systems, such as backup systems or analysis platforms.

[0040] PENDING means the task is in the queue waiting for scheduling; it has not yet been assigned a worker thread and has not started execution. RUNNING means the task has been scheduled, assigned to a worker thread, and is performing I / O, reading and writing data; it is running. ABORTED means the task has been manually aborted; it is the final termination state of the task.

[0041] In the embodiments described in this specification, each directory is associated with a task queue. Each task record in the task queue contains information such as task ID, task type, target file path, and task status (PENDING / RUNNING / COMPLETED / FAILED). The system maintains a unified task index (task progress: number of completed files / total number of files) in the metadata of the directory tree to be processed, so as to quickly query active tasks under all subdirectories.

[0042] The process iterates through the active tasks in the operation tasks, marking tasks in the PENDING and / or RUNNING states as ABORTED. Specifically, for tasks in the PENDING state, they are directly removed from the queue and marked as ABORTED (the user can manually retry later); for tasks in the RUNNING state, the abort_flag of the task context is set to true. The worker thread periodically checks this flag during its execution loop. If abort_flag = true is detected, then: the current data operation is stopped; existing file locks and resources are released; the task state is updated to ABORTED; and a termination log is recorded.

[0043] It should be noted that during the de-layering process, when any new flush / export task is submitted, the interception check interface is first called. If the check returns False, the task scheduling module rejects the task submission and returns the error code ERR_DIRECTORY_DETIERING_IN_PROGRESS and the message "The directory is currently undergoing de-layering operation and cannot submit flush / export tasks at this time".

[0044] Step S207: Traverse the list of directories to be processed, perform preheating to generate a preheating list, perform state transitions on the files in the preheating list, update the file states in the preheating list, and clear the hierarchical markers.

[0045] In this embodiment of the specification, the step of traversing the list of directories to be processed, performing preheating to generate a preheating list, performing state transitions on the files in the preheating list, updating the file states in the preheating list, and clearing the hierarchical markers specifically includes: Traverse the list of directories to be processed, identify files in remote storage or mixed status, and generate the preheating list. Based on the preheating list, files whose metadata current state is non-LOCAL are determined as target files, using files as the granularity. The data of the target file in the remote storage state is pulled back to the local storage. After the data integrity verification is passed, the storage state identifier of the target file is atomically updated to the local storage state, and the hierarchical mark is cleared.

[0046] In the embodiments of this specification, the step of pulling back the data of the target file in the remote storage state to the local storage state, and atomically updating the storage state identifier of the target file to the local storage state and clearing the hierarchical mark after the data integrity verification passes, specifically includes: Data from a target file that is in remote storage is pulled back to local storage, and data integrity is verified on the data pulled back to local storage. In response to the successful data integrity verification, the storage status identifier of the target file is updated to the local storage status, and the hierarchical marker is cleared; In response to a data integrity verification failure, the storage status identifier of the target file remains unchanged, and the verification failure information is recorded.

[0047] In this embodiment of the specification, the step of pulling back the data of the target file in the remote storage state to the local storage and performing data integrity verification on the data pulled back to the local storage specifically includes: In response to the target file's data volume exceeding a preset data volume threshold, the target file's data is retrieved in blocks and written to the local storage using a range download and streaming write method. Calculate the checksum of the data retrieved to the local storage, and compare the checksum with the original checksum recorded in the target file to perform data integrity verification.

[0048] In the embodiments described in this specification, the method further includes: Based on the preset concurrency configuration, multiple worker threads are scheduled to concurrently execute the data pullback; If, during the data pullback period, an access request for a target file that has not yet completed data pullback is received, a real-time data pullback of the target file is triggered. During the data pullback, a new file write request for the target directory is received. The newly written file is stored in local storage in a local storage state and does not participate in the hierarchical association attribute.

[0049] In one embodiment, the metadata for each file stored in the prediction list includes: file_path: file path; file_size: file size; state: file state (LOCAL / S3 / MIX); tiering_bucket: associated S3 bucket; tiering_key: object key in S3; checksum: data checksum (e.g., MD5, SHA-256); tiered: whether it has a hierarchical association (boolean value).

[0050] If state = LOCAL: The file data is already local and has no hierarchical association, so no processing is needed; simply mark it as "skip" and update the progress. If state = S3 or state = MIX, the data of the target file in remote storage will be pulled back to local storage. The local file's data verification value is then compared with the original verification value stored in the metadata. If they match, the file state is atomically updated to LOCAL. After a successful file state update, the system decrements the pending_tiered_count counter of the corresponding directory by 1 (atomic operation). If the counter drops to 0, the directory's is_tiered flag can be updated to false (this operation is optional and can also be completed by a unified check process). If they do not match, the file is marked as failed, the reason for failure is recorded as "data verification failed," and the processing result is input into the result collector, specifically including: file path, original state, target state, processing status (success / failure / skip), reason for failure (if any), processing time, etc.

[0051] Step S209: Determine whether the list of directories to be processed has been completely de-layered. If the list of directories to be processed has been completely de-layered, then clear the interception flags set in the list of directories to be processed and restore task scheduling.

[0052] To determine whether all directories have completed de-tiering, the system maintains a `pending_tiered_count` counter in each directory, recording the number of files still in the S3 / mixed state. When `pending_tiered_count` drops to 0, the directory can be marked as "de-tiered". If the `pending_tiered_count` of all directories is 0, then the overall de-tiering is complete.

[0053] If the list of pending directory trees has not been completely de-layered, a new directory de-layering request will be initiated for the list of failed files until the list of pending directory trees has been completely de-layered.

[0054] Because this manual uses a file-by-file processing granularity, it allows for the failure of some file conversions without affecting successfully converted files. The system returns the operation results to the user, including: the number of successfully converted files; the number of failed files and a list of failed files (path, reason for failure); and the current hierarchical status of each directory (fully de-hierarchized / partially de-hierarchized / not yet started). After addressing the issues based on the failure list (such as network recovery, storage space expansion, etc.), the user can re-initiate a de-hierarchical request for the incomplete parts.

[0055] It should be noted that, in actual execution, the following approach can also be adopted: If the de-layering of the pending directory tree list is partially completed after traversal, the interception flags set in the de-layered pending directory tree list can be cleared first, and task scheduling can be restored. For the un-de-layered directories in the pending directory tree list, the directory de-layering request can be re-initiated manually.

[0056] To further understand the directory de-layering method provided in the embodiments of this specification, the following will further explain it in conjunction with the state evolution of directories and files.

[0057] The initial state of the file ( ): The target directory / data / project_a is a hierarchical directory (is_tiered = true), associated with the S3 storage bucket old-bucket.

[0058] File status distribution in the directory: file1.txt: LOCAL (local hotspot data); file2.log: S3 (already flashed to S3, only metadata is available locally); file3.db: MIX (partial local caching, complete data on the remote end); The subdirectory / data / project_a / backup / inherits hierarchical attributes.

[0059] file4.bak:S3.

[0060] After de-layering is started, task interception ( ): The system setting block_flush_export = true means that all flush / export tasks are aborted or blocked.

[0061] Warm-up and file-by-file conversion ( file2.log: Data retrieval from S3 complete → Status changed to LOCAL, tiered = false; file3.db: Data retrieval from S3 complete → Status changed to LOCAL, tiered = false; file4.bak: Data retrieval from S3 complete → Status changed to LOCAL, tiered = false; Each file is converted independently, and unprocessed layered files may still exist in the directory / data / project_a (if any).

[0062] Directory attribute update ( The system checks the pending_tiered_count in the / data / project_a directory: Assuming all files in this directory have been processed, pending_tiered_count = 0, and the directory is_tiered is updated to false; The subdirectory / data / project_a / backup / is updated similarly.

[0063] intermediate state ( (Optional scenarios) If a file (such as file2.log) fails to be converted due to S3 being temporarily unavailable: file2.log remains in state S3, with tiered = true; Other files have been successfully converted to LOCAL; The pending_tiered_count of the directory / data / project_a is greater than 0, and the directory is_tiered remains true; The directory is in an intermediate state of "partially hierarchical". Users can later re-de-hierarchize file2.log.

[0064] Final state ( After all files are successfully converted, all files have state = LOCAL and tiered = false, all directories have is_tiered = false, and the entire directory tree is restored to a normal directory tree.

[0065] To facilitate traceability and auditing, this manual generates an audit log record each time the layer is delayered. The audit log record includes the following fields: operation unique identifier, operator username, operation type, operation initiation time, target directory path, whether recursive, concurrency configuration, total number of files, number of existing LOCAL state files, number of files converted from remote storage state, number of files converted from MIX state, number of successfully converted files, number of failed files, list of failed files (path + reason), total operation time (milliseconds), and final operation status (SUCCESS / PARTIAL_SUCCESS / FAILED).

[0066] The directory de-stratification method of the tiered storage system provided in this specification ensures that all data exists completely in local storage before the directory is restored to a normal directory by forcibly performing a full warm-up before removing the tiered attributes. This eliminates the risk of data inaccessibility due to disconnection of the S3 interface, thereby improving data security. During the warm-up period, users access incomplete files and fetch them in real time as needed. Newly written files are directly stored in the Local state. The de-stratification operation is imperceptible to normal business read and write operations, ensuring business continuity during storage backend migration. By automatically intercepting flush / export tasks, data corruption or inconsistency caused by concurrent tasks changing file states during the warm-up process is avoided, thus ensuring task consistency. All nested subdirectories are automatically processed to ensure that the tiered attributes of the entire directory tree are completely removed, eliminating any lingering issues of inconsistent tiered attributes between parent and child directories. Configurable multi-threaded / multi-process concurrent warm-up is supported, allowing users to dynamically adjust the concurrency based on network bandwidth and system resources, accelerating large-scale data migration.

[0067] The above describes in detail a method for de-stratifying a directory in a hierarchical storage system. Correspondingly, this specification also provides a system for de-stratifying a directory in a hierarchical storage system, such as... Figure 3 As shown. Figure 3 This is a schematic diagram of a directory de-hierarchy system for a hierarchical storage system provided in an embodiment of this specification. The directory de-hierarchy system includes: Get module 301, get directory de-layering request; Preprocessing module 303 performs preprocessing based on the directory de-hierarchy request to generate a list of directory trees to be processed; The interception module 305 performs task conflict detection and interception on the list of directories to be processed, and sets an interception flag in the list of directories to be processed. The state transition module 307 traverses the list of directories to be processed, performs preheating to generate a preheating list, performs state transitions on the files in the preheating list, updates the file status in the preheating list, and clears the hierarchical markers. The directory attribute processing module 309 determines whether the list of directory trees to be processed has been completely de-layered. If the list of directory trees to be processed has been completely de-layered, the interception flags set in the list of directory trees to be processed are cleared and task scheduling is restored.

[0068] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0069] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for apparatus, electronic devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0070] The apparatus, electronic device, and non-volatile computer storage medium and method provided in the embodiments of this specification are corresponding. Therefore, the apparatus, electronic device, and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, electronic device, and non-volatile computer storage medium will not be repeated here.

[0071] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0072] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0073] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0074] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0075] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0076] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0077] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0078] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0079] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0080] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0081] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0082] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0083] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside on local and remote computer storage media, including storage devices.

[0084] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0085] The above description is merely an embodiment of this specification and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. A method for de-hierarchizing a directory in a hierarchical storage system, characterized in that, The directory de-hierarchy method includes: Get directory de-layering request; Based on the directory de-hierarchy request, preprocessing is performed to generate a list of directory trees to be processed; Perform task conflict detection and interception on the list of directories to be processed, and set interception flags in the list of directories to be processed; Traverse the list of directories to be processed, perform preheating to generate a preheating list, perform state transitions on the files in the preheating list, update the file states in the preheating list, and clear the hierarchical markers; Determine whether the list of directory trees to be processed has been completely de-layered. If the list of directory trees to be processed has been completely de-layered, then clear the interception flags set in the list of directory trees to be processed and restore task scheduling.

2. The directory de-layering method as described in claim 1, characterized in that, The preprocessing based on the directory de-hierarchy request to generate a list of directory trees to be processed specifically includes: Based on the directory de-layering request, the target hierarchical directory with hierarchical association attributes is determined; The target hierarchical directory and its nested subdirectories are traversed to generate a list of directory trees to be processed.

3. The directory de-layering method as described in claim 2, characterized in that, The step of traversing the target hierarchical directory and its nested subdirectories to generate a list of directory trees to be processed specifically includes: Using the target hierarchical directory as the root node, the nested subdirectories of the target hierarchical directory are traversed to generate the list of directories to be processed. The metadata of the list of directories to be processed includes: directory path, hierarchical identifier, associated remote storage name, remote storage path prefix, total number of files, and number of files initially in remote / mixed state.

4. The directory de-layering method as described in claim 1, characterized in that, The step of performing task conflict detection and interception on the list of directories to be processed, and setting interception flags in the list of directories to be processed, specifically includes: Query the operation tasks under the pending directory tree list, the operation tasks include brushing tasks and export tasks; Traverse the active tasks in the operation tasks and mark the tasks that are in the PENDING and / or RUNNING states as ABORTED. Set an interception flag in the metadata of the root directory of the pending directory tree list, and synchronize the interception flag to all subdirectories in the pending directory tree list.

5. The directory de-layering method as described in claim 1, characterized in that, The process of traversing the list of directories to be processed, performing preheating to generate a preheating list, performing state transitions on the files in the preheating list, updating the file states in the preheating list, and clearing hierarchical markers specifically includes: Traverse the list of directories to be processed, identify files in remote storage or mixed status, and generate the preheating list. Based on the preheating list, files whose metadata current state is non-LOCAL are determined as target files, using files as the granularity. The data of the target file in the remote storage state is pulled back to the local storage. After the data integrity verification is passed, the storage state identifier of the target file is atomically updated to the local storage state, and the hierarchical mark is cleared.

6. The directory de-layering method as described in claim 5, characterized in that, The process of pulling back the data of the target file, which is currently in remote storage, to local storage, and atomically updating the storage status identifier of the target file to the local storage status after the data integrity verification passes, and clearing the hierarchical marker, specifically includes: Data from a target file that is in remote storage is pulled back to local storage, and data integrity is verified on the data pulled back to local storage. In response to the successful data integrity verification, the storage status identifier of the target file is updated to the local storage status, and the hierarchical marker is cleared; In response to a data integrity verification failure, the storage status identifier of the target file remains unchanged, and the verification failure information is recorded.

7. The directory de-layering method as described in claim 6, characterized in that, The step of pulling back the data of the target file that is in remote storage to local storage and performing data integrity verification on the data pulled back to local storage specifically includes: In response to the target file's data volume exceeding a preset data volume threshold, the target file's data is retrieved in blocks and written to the local storage using a range download and streaming write method. Calculate the checksum of the data retrieved to the local storage, and compare the checksum with the original checksum recorded in the target file to perform data integrity verification.

8. The directory de-layering method as described in claim 5, characterized in that, The method further includes: Based on the preset concurrency configuration, multiple worker threads are scheduled to concurrently execute the data pullback; If, during the data pullback period, an access request for a target file that has not yet completed data pullback is received, a real-time data pullback of the target file is triggered. During the data pullback, a new file write request for the target directory is received. The newly written file is stored in local storage in a local storage state and does not participate in the hierarchical association attribute.

9. The directory de-layering method as described in claim 1, characterized in that, The method further includes: If the list of pending directory trees has not been completely de-layered, a new directory de-layering request will be initiated for the list of failed files until the list of pending directory trees has been completely de-layered.

10. A directory de-hierarchy system for a hierarchical storage system, characterized in that, The directory de-hierarchy system includes: Get the module, get the directory de-layering request; The preprocessing module performs preprocessing based on the directory de-hierarchy request to generate a list of directory trees to be processed; The interception module performs task conflict detection and interception on the list of directories to be processed, and sets an interception flag in the list of directories to be processed. The state transition module traverses the list of directories to be processed, performs preheating to generate a preheating list, performs state transitions on the files in the preheating list, updates the file status in the preheating list, and clears the hierarchical markers. The directory attribute processing module determines whether the list of directory trees to be processed has been completely de-layered. If the list of directory trees to be processed has been completely de-layered, the interception flags set in the list of directory trees to be processed are cleared and task scheduling is restored.