Data storage management method, device and equipment and readable storage medium

Abstract

The invention provides a data storage management method, a data storage management device, data storage management equipment and a readable storage medium. The access behavior of each storage fragment under a specified directory is counted in a statistical period before the data migration operation of the specified directory is triggered, and when the data migration operation is triggered at the end of the statistical period, all the storage fragments are used as to-be-migrated data aiming at files with read-write operations in the statistical period; and for the file only having the read operation in the statistical period, the read storage fragment is used as the to-be-migrated data, so that when the to-be-migrated data is migrated to the high-performance layer storage area, redundant data migration is reduced; the problem that the storage space of a high-performance layer is excessively occupied due to the fact that a whole-file full-amount migration mode is adopted in the related technology is effectively solved, and unnecessary storage resource consumption is reduced.

Description

Technical Field

[0001] This application relates to the field of distributed file storage technology, and in particular to a data storage management method, apparatus, device, and readable storage medium. Background Technology

[0002] Distributed file systems utilize hierarchical storage and employ different strategies to enable flexible file transfer between high-performance and low-performance storage areas, thereby achieving the goal of accelerating the response of hot data and reducing the storage cost of cold data.

[0003] In file migration scenarios from low-performance storage tiers to high-performance storage tiers, the current common approach is full file migration: after a data migration operation is triggered, for each file that matches the migration rules, all storage fragments corresponding to each file segment within that file are migrated to the high-performance storage tier. However, this full file migration approach has significant resource overhead issues. Full file migration results in a large amount of unaccessed data being redundantly loaded into the high-performance storage tier, consuming a significant amount of storage space and causing a rapid increase in the occupancy rate of high-cost storage media, thereby increasing the overall storage cost. Summary of the Invention

[0004] In view of this, in order to solve the above-mentioned technical problems, this application provides a data storage management method, apparatus, device and readable storage medium, which can effectively solve the problem that the existing methods result in large storage space consumption in the high-performance layer storage area.

[0005] Specifically, this application is implemented through the following technical solution: According to a first aspect of the embodiments of this application, a data storage management method is provided, applied to a distributed file system, wherein the data storage space within the distributed file system is divided into a high-performance layer storage area and a low-performance layer storage area; the method includes: When a data migration operation is triggered in a specified directory within the distributed file system, the following operations are performed to determine the data to be migrated in that specified directory: If all storage shards corresponding to any file identifier in the specified directory meet the first condition, then the data carried by all storage shards corresponding to that file identifier will be used as the data to be migrated. The first condition includes that at least one storage shard has been written to and at least one storage shard has been read to within the statistical period. The statistical period is a preset time window and its end time is aligned with the trigger time of the data migration operation. If it is detected that all storage fragments corresponding to any file identifier in the specified directory meet the second condition, then the data carried by the storage fragments that have only performed read operations among all storage fragments corresponding to that file identifier will be regarded as the data to be migrated; the second condition includes that no write operations have been performed on any storage fragments during the statistical period and at least one storage fragment has performed a read operation. The identified data to be migrated is moved from the low-performance layer storage area to the high-performance layer storage area.

[0006] In some embodiments, a popularity statistics set is maintained for the specified directory; the popularity statistics set is stored in the form of access records; each access record includes at least a file identifier and an operation type; the operation type includes at least a read flag, a write flag, and a file migration flag; A read flag in any access record indicates that none of the storage segments corresponding to the file identifier in that access record have been written to, and at least one storage segment has been read to. A write flag in any access record indicates that none of the storage segments corresponding to the file identifier in that access record have been read, and at least one storage segment has been written. A file migration marker in any access record indicates that at least one storage segment corresponding to the file identifier in that access record has been written to and at least one storage segment has been read from.

[0007] In some embodiments, detecting that all storage fragments corresponding to any file identifier in the specified directory satisfy the first condition includes: The heat statistics set contains access records corresponding to the file identifier, and the operation type in the access record is file migration marker.

[0008] In some embodiments, detecting that all storage fragments corresponding to any file identifier in the specified directory satisfy the second condition includes: When the access record corresponding to the file identifier exists in the heat statistics set, and the operation type in the access record is a read marker when the statistics end event corresponding to the specified directory is detected.

[0009] In some embodiments, before the data migration operation of the specified directory is triggered, the method further includes: If a read operation is received for any file in the specified directory within the statistical period, query whether there is a first access record in the popularity statistics set that matches the file identifier carried by the read operation. If it exists, then if the operation type in the first access record is a write flag, update the write flag to the file migration flag; If it does not exist, a new access record is created in the heat statistics set; the file identifier in the new access record is the file identifier carried by the read operation, and the operation type is read tag.

[0010] In some embodiments, each access record further includes a set of read storage fragments; the set of read storage fragments is used to record the identifiers of the storage fragments that have performed read operations corresponding to the file identifiers in the access record; the method further includes: If a read operation is received for any file in the specified directory within the statistical period, the identifier of each storage segment read by the read operation is stored in the set of read storage segments in the first access record or the new access record; Specifically, during the process of storing the identifier of each storage segment read by the read operation into the set of read storage segments, if the set of read storage segments already includes the identifier of the storage segment, the read operation count corresponding to the storage segment is updated; if the set of read storage segments does not include the identifier of the storage segment, the identifier of the storage segment is recorded in the set of read storage segments and the read operation count is initialized.

[0011] In some embodiments, the data carried by the storage fragments that have only undergone read operations among all storage fragments corresponding to the file identifier is taken as the data to be migrated, including: Query the access records that match the file identifier from the heat statistics set, and obtain the number of read operations corresponding to each storage segment in the read storage segment set from the access records; Each storage fragment in the set of read storage fragments whose number of read operations exceeds a set threshold is considered as data to be migrated.

[0012] In some embodiments, before the data migration operation of the specified directory is triggered, the method further includes: If a write operation is received for any file in the specified directory within the statistical period, the system queries the popularity statistics set to see if there is a second access record that matches the file identifier carried by the write operation. If it exists, and the operation type in the second access record is a read marker, the read marker is updated to the file migration marker. If it does not exist, a new access record is created in the popularity statistics set. The file identifier in the new access record is the file identifier carried by the write operation, and the operation type is a write marker.

[0013] According to a second aspect of the embodiments of this application, a data storage management apparatus is provided, applied to a distributed file system, wherein the data storage space within the distributed file system is divided into a high-performance layer storage area and a low-performance layer storage area; the apparatus includes: The migration data determination module is configured to perform the following operations to determine the data to be migrated in the specified directory when a data migration operation in the specified directory is triggered in the distributed file system: If it is detected that all storage shards corresponding to any file identifier in the specified directory meet a first condition, then the data carried by all storage shards corresponding to that file identifier is taken as the data to be migrated; the first condition includes that at least one storage shard has been written to and at least one storage shard has been read to within a statistical period; the statistical period is a preset time window and its end time is aligned with the trigger time of the data migration operation; If it is detected that all storage shards corresponding to any file identifier in the specified directory meet a second condition, then the data carried by the storage shard that has only been read to within the storage shards corresponding to that file identifier is taken as the data to be migrated; the second condition includes that no write operation has been performed on any storage shards within the statistical period and at least one storage shard has been read to within the statistical period. The migration operation execution module is configured to migrate the determined data to be migrated from the low-performance layer storage area to the high-performance layer storage area.

[0014] In some embodiments, a popularity statistics set is maintained for the specified directory; the popularity statistics set is stored in the form of access records; each access record includes at least a file identifier and an operation type; the operation type includes at least a read flag, a write flag, and a file migration flag; A read flag in any access record indicates that none of the storage segments corresponding to the file identifier in that access record have been written to, and at least one storage segment has been read to. A write flag in any access record indicates that none of the storage segments corresponding to the file identifier in that access record have been read, and at least one storage segment has been written. A file migration marker in any access record indicates that at least one storage segment corresponding to the file identifier in that access record has been written to and at least one storage segment has been read from.

[0015] In some embodiments, detecting that all storage fragments corresponding to any file identifier in the specified directory satisfy the first condition includes: The heat statistics set contains access records corresponding to the file identifier, and the operation type in the access record is file migration marker.

[0016] In some embodiments, detecting that all storage fragments corresponding to any file identifier in the specified directory satisfy the second condition includes: When the access record corresponding to the file identifier exists in the heat statistics set, and the operation type in the access record is a read marker when the statistics end event corresponding to the specified directory is detected.

[0017] In some embodiments, before the data migration operation of the specified directory is triggered, the apparatus further includes a read operation processing module configured to: If a read operation is received for any file in the specified directory within the statistical period, the system queries the popularity statistics set to see if there is a first access record that matches the file identifier carried by the read operation. If it exists, and the operation type in the first access record is a write flag, the write flag is updated to the file migration flag. If it does not exist, a new access record is created in the popularity statistics set. The file identifier in the new access record is the file identifier carried by the read operation, and the operation type is a read flag.

[0018] In some embodiments, each access record further includes a set of read storage fragments; the set of read storage fragments is used to record the identifiers of the storage fragments that have undergone read operations corresponding to the file identifiers in the access record; the apparatus further includes a read fragment statistics module, configured to: If a read operation is received for any file in the specified directory within the statistical period, the identifier of each storage segment read by the read operation is stored in the read storage segment set in the first access record or the new access record; wherein, during the process of storing the identifier of each storage segment read by the read operation into the read storage segment set, if the read storage segment set already includes the identifier of the storage segment, the read operation count corresponding to the storage segment is updated; if the read storage segment set does not include the identifier of the storage segment, the identifier of the storage segment is recorded in the read storage segment set and the read operation count is initialized.

[0019] In some embodiments, when the migration data determination module is configured to use data carried by storage fragments that have only undergone read operations among all storage fragments corresponding to the file identifier as the data to be migrated, it includes: The system queries the access records that match the file identifier from the heat statistics set, and obtains the number of read operations corresponding to each storage fragment in the read storage fragment set from the access records; each storage fragment in the read storage fragment set whose number of read operations exceeds a set threshold is regarded as data to be migrated.

[0020] In some embodiments, before the data migration operation of the specified directory is triggered, the apparatus further includes a write operation processing module configured to: If a write operation is received for any file in the specified directory within the statistical period, the system queries the popularity statistics set to see if there is a second access record that matches the file identifier carried by the write operation. If it exists, and the operation type in the second access record is a read marker, the read marker is updated to the file migration marker. If it does not exist, a new access record is created in the popularity statistics set. The file identifier in the new access record is the file identifier carried by the write operation, and the operation type is a write marker.

[0021] According to a third aspect of the embodiments of this application, an electronic device is provided, the electronic device comprising: a memory and a processor; the memory being used to store a computer program; the processor being used to execute the above-described data storage management method by invoking the computer program.

[0022] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, wherein the program, when executed by a processor, implements the above-described data storage management method.

[0023] The technical solutions provided in this application embodiment may include the following beneficial effects: In the technical solution provided in this application, when a data migration operation in a specified directory is triggered, the storage segments within a file are classified and identified based on the read and write operation records of each storage segment in the specified directory within a preset statistical period. For files that have both read and write operations within the statistical period, the data carried by all storage segments included in the file is taken as the data to be migrated. For files that only have read operations within the statistical period, the data carried by the storage segment corresponding to the file that is only actually read is taken as the data to be migrated. This reduces the amount of redundant data to be migrated to the high-performance storage layer, effectively controls the space occupancy rate of high-cost storage media, and reduces the overall storage cost of the system.

[0024] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Furthermore, no embodiment in this application needs to achieve all the effects described above. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0026] Figure 1A This is an exemplary diagram illustrating how a large file is divided into multiple file segments and corresponding storage partitions. Figure 1B This is an exemplary migration flowchart under a periodic migration strategy for a specified directory configuration in related technologies. Figure 2A This is a schematic diagram of a data storage management method according to an exemplary embodiment of this application; Figure 2B This is a schematic diagram illustrating an exemplary embodiment of this application of triggering a statistical start event and a statistical end event; Figure 2C This is a schematic diagram illustrating another method for triggering statistical start and statistical end events, as shown in an exemplary embodiment of this application. Figure 3A This is a schematic diagram illustrating the structure of a heat statistics set according to an exemplary embodiment of this application; Figure 3B This is a flowchart illustrating an exemplary embodiment of the present application of a step for updating the heat statistics set of read operations received for any file in the specified directory within the current statistical period; Figure 3C This is a flowchart illustrating an exemplary embodiment of the present application of a step for updating a set of hot statistics on write operations received for any file in a specified directory within the current statistical period; Figure 4A This is an overall flowchart illustrating a data storage management method according to an exemplary embodiment of this application; Figure 4B This is a schematic diagram illustrating the processing of a single I / O operation within a statistical period, as shown in an exemplary embodiment of this application; Figure 5 This is a schematic diagram of the structure of a data storage management device shown in an exemplary embodiment of this application; Figure 6 This is a hardware schematic diagram of an electronic device illustrated in an exemplary embodiment of this application. Detailed Implementation

[0027] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another.

[0028] Before explaining the data storage management method provided in this application, a brief explanation of some of the terms or concepts involved in this application will be provided: A distributed file system is a system architecture that efficiently and reliably stores massive amounts of data across multiple physical servers. It divides a large file into several fixed-length or variable-length segments (often called chunks) and distributes these segments across different storage nodes. Through a unified namespace, centralized or distributed metadata services, and standardized network access protocols, this distributed file system presents itself as a logical global file system, allowing users to transparently read and write files across machines as if they were operating a local directory. To ensure data reliability and high availability, each file segment typically employs EC (Erasure Coding) or multi-replica redundancy strategies for data protection. This ensures no data loss and no service interruption in the event of a single point of failure or even multiple server failures, achieving seamless fault tolerance for upper-layer business processes.

[0029] EC: The file segment to be stored is divided into k raw data fragments, and m parity fragments are generated based on the k raw data fragments using a specific mathematical encoding algorithm, thus forming a total of k+m storage fragments. The k+m storage fragments are distributed and stored on different physical nodes. At any given time, the number of available storage fragments (i.e., without failure) is no less than k, and regardless of whether these storage fragments come from raw data fragments or parity fragments, the complete original file segment can be reconstructed losslessly using a decoding algorithm.

[0030] Multiple replicas: The same file segment is completely copied into multiple copies to form multiple storage fragments, and these multiple storage fragments are stored on different nodes or disks respectively; if any one copy is damaged, the distributed file system can still provide read and write services using file copies on other available nodes or disks, thereby improving storage reliability.

[0031] Tiered storage: The data storage space within a distributed file system is divided into a high-performance tier storage area and a low-performance tier storage area. Data in the distributed file system is divided into hot and cold data based on access frequency, performance requirements, or cost sensitivity. According to preset migration strategy rules, hot data is migrated from the low-performance tier storage area to the high-performance tier storage area, and cold data is migrated from the high-performance tier storage area to the low-performance tier storage area. This satisfies users' diverse needs for file processing performance, storage capacity, and cost, thereby achieving the goals of rationally utilizing storage space, improving storage system access performance, and reducing overall deployment costs.

[0032] High-performance storage tier: Offers better read / write response speeds and concurrent processing capabilities. Typically uses high-performance solid-state drives (SSDs) as the primary storage medium and is usually connected via SAS (Serial Attached SCSI), SATA (Serial Advanced Technology Attachment), or NVMe (Non-Volatile Memory Express) interfaces. However, it has a higher unit storage cost and is typically used to store frequently accessed, latency-sensitive hot data.

[0033] Low-performance tier storage: This typically uses mechanical hard drives (HDDs) as the storage medium, usually connected via SAS or SATA interfaces. Compared to SSDs, HDDs have a rotating disk and a head seek mechanism, resulting in higher I / O latency and lower random read / write performance, but lower cost per unit capacity. The low-performance tier storage is suitable for storing cold or warm data with low access frequency and insensitivity to latency, effectively reducing overall storage costs while ensuring basic availability.

[0034] See Figure 1A An exemplary diagram illustrates the division of a large file into multiple file segments and corresponding storage shards. A large file is divided into multiple file segments of a fixed size (e.g., 64MB). Each file segment is stored redundantly using an EC (4+2) redundancy strategy. A file segment is divided into 4 data shards and 2 parity shards are generated, forming 6 storage shards. These 6 storage shards are stored on different storage nodes.

[0035] Based on this, existing distributed file systems maintain file access paths and permissions while utilizing tiered storage and different strategies to flexibly transfer files between high-performance and low-performance storage layers. This achieves the goal of accelerating hot data response and reducing storage costs for cold data. For file migration from the high-performance to the low-performance storage layer, the distributed file system allows users to customize cold data identification rules by combining multiple attributes such as file name, size, creation time, modification time, and last access time, based on a specified directory. All storage shards corresponding to each file that meets these cold data identification rules will be automatically migrated to the low-performance storage layer, freeing up high-cost, high-bandwidth high-performance storage resources. Upper-layer applications are completely unaware of the file access paths, naming conventions, and permissions.

[0036] For migrations from low-performance storage tiers to high-performance storage tiers (i.e., performance improvement scenarios), users can also configure policies based on directories to set hot data identification rules for files that are expected to be frequently accessed or modified in the near future, combined with the above-mentioned multi-dimensional attributes. All storage shards corresponding to each file that meets the hot data identification rules will be migrated to the high-performance storage tier in advance, so that subsequent read and write operations can fully utilize the low latency and high throughput advantages brought by high-speed media such as SSDs. The business side also does not need to be aware of the change in the underlying storage location.

[0037] For the file migration scenario from the low-performance storage layer to the high-performance storage layer, the relevant technologies can configure a one-time migration policy (for emergency or specific task triggering) or a periodic migration policy (such as daily or weekly regular execution) for a specified directory. When the migration policy is triggered, the distributed file system will automatically scan the files in the specified directory that meet the migration conditions and migrate all storage shards corresponding to the file to ensure that hot data is in place in a timely manner, taking into account both performance response and operation and maintenance automation.

[0038] See Figure 1B The flowchart illustrating an exemplary periodic migration strategy describes a process where, for a specified directory configured with this strategy, when data migration operations are periodically triggered for that directory, each file within the specified directory is sequentially checked against a preset file migration rule. If a file meets the rule, all storage shards of that file are migrated to the high-performance tier storage area, until all files in the specified directory have been traversed. This periodic rule can be executed monthly, weekly, daily, or at intervals of several hours to meet flexible scheduling needs arising from varying business peaks and busts. The file migration rule can be determined by combining multiple attributes such as file creation time, modification time, last access time, file size, and file name.

[0039] As can be seen, in file migration scenarios from low-performance tier storage to high-performance tier storage, the existing approach involves migrating all storage fragments of all file segments for each file that matches the migration rules after the data migration operation is triggered. After the migration is complete, all read and write requests are directly written to the high-performance tier storage. However, this full file migration method has significant resource overhead, consuming a large amount of storage space in the high-performance tier storage, leading to a rapid increase in the occupancy rate of high-cost storage media, and consequently increasing the overall storage cost. Under large-scale deployments or highly redundant configurations, this space overhead may quickly encroach on the capacity of the high-performance tier, limiting the scale of hot data the system can handle, thus violating the design objectives of tiered storage: on-demand allocation and cost optimization.

[0040] For example, suppose directory dir1 uses an EC 4+2 redundancy strategy, the user configures a migration task to be performed every Monday at 22:00, the file migration rule is "last file access time ≤ 7 days", the directory has 100 files, the size of each file is 2 GB, when the data migration operation is triggered at 22:00 on Monday, files file1 ~ file10 are detected to meet the file migration rule, then the actual space occupied by the file after migration in the high-performance layer storage area is: 10 × 2 GB × (4 + 2) / 4 = 30 GB. Subsequent reads / writes for file1 ~ file10 are all completed in the high-performance layer storage area, which significantly improves performance but consumes more space in the high-performance layer.

[0041] Similarly, assuming directory dir2 uses a three-replica redundancy strategy, the user configures a migration task every Wednesday at 22:00, and the file migration rule is "last file access time ≤ 7 days", the directory has a total of 200 files, and the size of each file is 5 GB. When file1 to file50 are detected to meet the file migration rule at Wednesday at 22:00, the actual space occupied by migrating the file to the high-performance layer storage area is: 50 × 5 GB × 3 = 750 GB. Subsequent reads / writes for file1 to file50 are all completed in the high-performance layer, which significantly improves performance, but the high-performance layer consumes more space.

[0042] In view of this, this application provides a data storage management method for the scenario of migrating data from the low-performance storage layer to the high-performance storage layer within the distributed file system, applied to the aforementioned distributed file system. This method dynamically identifies file access patterns in a specified directory within a statistical period before the data migration operation is triggered, and filters file storage fragments based on different access behaviors. Therefore, when the data migration operation is triggered at the end of the statistical period, data with active access characteristics or potential performance requirements is selected as the data to be migrated. Specifically, for files with both read and write operations within the statistical period, all their storage fragments are included in the migration scope to ensure data consistency and access efficiency. For files with only read operations within the statistical period, only the read storage fragments are migrated, avoiding redundant data movement. This effectively solves the problem of excessive high-performance storage space occupation caused by the full file migration method in related technologies, reducing unnecessary storage resource consumption. While ensuring critical data access performance and system response efficiency, it optimizes the utilization efficiency of high-performance storage resources, thereby reducing the overall storage cost and operational overhead of the distributed file system.

[0043] See Figure 2A The illustrated schematic diagram shows the steps of a data storage management method. The data storage management method provided in this embodiment may include at least the following steps: S201, if a data migration operation in a specified directory of the distributed file system is triggered, perform the following operations to determine the data to be migrated in the specified directory: If all storage shards corresponding to any file identifier in the specified directory meet the first condition, then the data carried by all storage shards corresponding to that file identifier will be used as the data to be migrated. The first condition includes that at least one storage shard has been written to and at least one storage shard has been read to within the statistical period. The statistical period is a preset time window and its end time is aligned with the trigger time of the data migration operation. If it is detected that all storage fragments corresponding to any file identifier in the specified directory meet the second condition, then the data carried by the storage fragments that have only performed read operations among all storage fragments corresponding to that file identifier will be regarded as the data to be migrated; the second condition includes that no write operations have been performed on any storage fragments during the statistical period and at least one storage fragment has performed a read operation. The designated directory refers to the target directory in the distributed file system that has been pre-marked by the user or administrator and requires data popularity identification and tiered migration. In this embodiment, a one-time data migration strategy or a periodic migration strategy can be configured for the designated directory. The one-time data migration strategy is suitable for snapshot-style analysis and migration of data status at a specific point in time, such as manually triggering a full-scale hot and cold data tiering operation during off-peak business periods. The periodic migration strategy automatically triggers the migration process at preset time intervals (such as every 6 hours, daily, or weekly). Regardless of the migration strategy used, when each data migration operation is triggered, the data popularity is dynamically evaluated based on the storage shard access behavior in the most recent statistical period, and the data to be migrated is selected accordingly.

[0044] A data migration operation refers to the operation of migrating data from the low-performance storage layer of a distributed file system to the high-performance storage layer. The data migration operation is triggered when the current statistical period ends, and the distributed file system needs to determine the data to be migrated in the specified directory based on the storage shard access behavior (including read / write operation records) collected during the current statistical period, so as to promote the data to be migrated to the high-performance storage layer.

[0045] The statistical period represents a time window used to observe and record storage shard access behavior (including read and write operations) under a specified directory. This statistical period is a preset time window, and its end time is aligned with the trigger time of the data migration operation. That is, whenever a data migration operation is triggered, the current statistical period has just ended. The distributed file system performs a heat analysis based on the accumulated storage shard access behavior under the directory within this statistical period and determines the data to be migrated accordingly. In other words, the statistical period is not an arbitrary time period, but rather the most recent time window closely coupled with the data migration operation. When a periodic migration strategy is configured for the specified directory, the start time of this statistical period is determined by the completion time of the previous data migration operation or the system initialization time. The start and end times of this statistical period are aligned with the trigger time of the current data migration operation to ensure that the collected storage shard access behavior fully covers a complete decision-making cycle, avoiding data truncation or cross-cycle contamination.

[0046] For example, if a specified directory is configured with a periodic data migration strategy, and the statistical period is set to once every 24 hours, and the previous round of data migration operation completed the operation of migrating data from the low-performance tier storage area to the high-performance tier storage area at 2:00 AM on the same day, then the distributed storage system starts a new round of statistical period at 2:00 AM on the same day. This statistical period lasts for 24 hours and ends at 2:00 AM on the next day, triggering a data migration operation. Thus, after the data to be migrated is determined at 2:00 AM on the next day, the operation of migrating the data to be migrated from the low-performance tier storage area to the high-performance tier storage area is executed.

[0047] To facilitate internal system state management and event-driven scheduling, statistical start events and statistical end events can be introduced to identify the boundaries of the statistical period. The statistical start event represents the start signal of the current round of data popularity statistics, indicating the beginning of the current statistical period; the statistical end event represents the end signal of the current statistical period, indicating the closing of the statistical time window and triggering the data migration operation execution process.

[0048] Taking the aforementioned 24-hour cycle as an example, the system triggers a statistics start event at 2:00 AM on the same day to begin recording the read and write operations of each file storage shard in that directory; and triggers a statistics end event at 2:00 AM the following day, marking the end of this statistical cycle and triggering a data migration operation so that the distributed file system can begin to determine the data to be migrated and promote the data to be migrated to the high-performance storage layer.

[0049] Furthermore, the length of the statistical period can be dynamically adjusted according to actual business needs. Different statistical periods can be configured for the same specified directory under different business scenarios or system load characteristics. For example, a shorter statistical period can be used during periods of high load or drastic changes in access patterns to respond more quickly to hotspot changes; while a longer statistical period can be used during periods of low load or stable access to reduce migration overhead and system disturbance.

[0050] For different specified directories, independent statistical periods can be configured according to their business importance, data access frequency, service level agreement requirements, or performance requirements, thereby realizing fine-grained and differentiated storage resource scheduling strategies.

[0051] When configuring a one-time migration policy for a specified directory, for example, see... Figure 2B This example illustrates a schematic diagram of triggering statistical start and end events. A one-time migration policy is configured at time point t0, triggering the statistical start event at t0. Alternatively, if the effective time t1 of the one-time migration policy is specified, the statistical start event is triggered at time point t1. Correspondingly, if the data migration operation is triggered at time point t3, the statistical end event is triggered at time point t3. Alternatively, if the statistical period is specified as T, the statistical end event is triggered when the trigger time of the statistical start event has elapsed through the time window T. When the statistical end event is triggered, it indicates the end of the current statistical period. It is determined that the data migration operation for the specified directory has been triggered, and the distributed file system begins to determine the data to be migrated under that specified directory, migrating the data from the low-performance storage layer to the high-performance storage layer.

[0052] If a periodic migration policy is configured for a specified directory, see [link / reference]. Figure 2C The illustrative diagram illustrates another method for triggering statistical start and end events. A periodic migration strategy is configured at time point t0, triggering a statistical start event at t0. Alternatively, if the effective time t1 of the periodic migration strategy is specified, the statistical start event is triggered at time point t1. When the statistical period for the specified directory is T, a statistical end event is triggered at time point (t0+T) or (t1+T), simultaneously triggering a new statistical start event to indicate the start of a new statistical period. When the statistical end event is triggered, it indicates the end of the current statistical period, confirming that the data migration operation for the specified directory has been triggered.

[0053] The first condition defines a scenario where, within the time interval defined from the start of the statistics event to the end of the statistics event (i.e., the current statistical period), the distributed file system receives both a read operation and a write operation on the same file. This means that the same file experiences mixed access behavior of read and write operations within this entire time interval, indicating high activity or potentially critical data. Therefore, all storage shards of files meeting the first condition should be migrated to the high-performance layer to ensure consistency and access efficiency. Based on this, the first condition includes the existence of at least one storage shard undergoing a write operation and at least one storage shard undergoing a read operation within the statistical period. For example, if a first storage shard of any file undergoes a read operation and a second storage shard undergoes a write operation, the relationship between the first and second storage shards can be any of the following: (1) The first storage fragment and the second storage fragment are the same storage fragment, that is, the same storage fragment is both read from and written to within this time interval, for example Figure 1A The file segment shown is storage fragment D0 under c#3; (2) The first storage fragment and the second storage fragment are different storage fragments of the same file segment, for example Figure 1A The file segment shown in C#3 has storage fragments D0 and D2. One storage fragment, such as D0, is read from, and the other storage fragment, such as D2, is written to. (3) The first storage fragment and the second storage fragment are two storage fragments of different file segments of the same file, for example Figure 1A The storage fragment D0 under file segment c#3 and storage fragment D0 under another file segment c#79 are shown.

[0054] Based on the above, when a data migration operation in the specified directory is triggered, if all storage shards corresponding to any file identifier in the specified directory meet the first condition within the statistical period, then the data carried by each storage shard under each file segment of the large file pointed to by that file identifier will be used as the data to be migrated. For example, using... Figure 1A For example, each file segment corresponds to 6 storage shards, and 80 file segments correspond to 480 storage shards. During the statistical period, if at least one of the 480 storage shards is subjected to a read operation and at least one of the storage shards is subjected to a write operation, then the data carried by the 480 storage shards corresponding to the file will be used as the data to be migrated.

[0055] The second condition is used to define that: within the entire time interval from the start event to the end event of the statistics, the distributed file system only receives read operations on any file; that is, the same file only has read access behavior and no write operations occur during this entire time interval, and its access popularity is concentrated on a portion of the storage shards. To avoid the waste of resources caused by migrating redundant storage shards and low-access-frequency file segments to the high-performance layer storage space, this implementation step migrates the data carried by each storage shard that has undergone read operations corresponding to the file identifier that meets the second condition to the high-performance layer. This ensures the performance of hot reads while maximizing the utilization efficiency of high-performance layer storage resources and reducing overall storage costs. Based on this, the second condition is defined as including that within the time interval from the start event to the end event of the statistics, all storage shards corresponding to the file identifier have not undergone write operations, and at least one of them has undergone read operations.

[0056] For example, still using Figure 1A For example, each file segment corresponds to 6 storage shards, and 80 file segments correspond to 480 storage shards. During the statistical period, if no write operation is performed on any of the 480 storage shards, and at least one storage shard is subjected to a read operation, then the data carried by each storage shard subjected to the read operation is regarded as data to be migrated.

[0057] S202, the determined data to be migrated is migrated from the low-performance layer storage area to the high-performance layer storage area.

[0058] This implementation procedure completes the transparent migration of data between storage levels without changing the original access paths of users or applications. During and after the data migration, the data's access path, file identifier, permission attributes, and other metadata remain unchanged.

[0059] Distributed file systems employ an architecture that separates logical namespaces from physical storage locations. Users access data through a unified file path, which is actually mapped to storage shards on multiple backend physical storage nodes. The logical metadata of a file is centrally managed by a metadata service, while the physical data blocks / shards of the file are stored in either a low-performance or high-performance tier. When data migration is performed, only the location of the storage shards is moved, without modifying the file's logical path or other metadata. Therefore, users can still seamlessly access files through the original access path without being aware of changes to the underlying storage hierarchy.

[0060] Furthermore, after moving the data to be migrated from the low-performance tier storage area to the high-performance tier storage area, the distributed file system atomically updates the physical location pointer of each storage fragment corresponding to the migrated data, making it point to the specific storage location of the storage fragment in the high-performance tier storage area. Each access request for the migrated data is automatically routed to the high-performance tier storage area after the metadata is updated.

[0061] After migrating the data to be migrated from the low-performance tier storage area to the high-performance tier storage area, the data in the low-performance tier storage area can be retained to support data redundancy, disaster recovery, or rapid rollback to the original storage tier when access patterns change. Alternatively, after confirming that the data to be migrated has been completely written into the high-performance tier storage area, the data in the low-performance tier storage area can be asynchronously deleted to save storage space in the low-performance tier storage area. In practical applications, the retention or deletion strategy can be dynamically determined based on preset storage strategies, data importance levels, cost optimization goals, or user configuration. For example, for critical business data, dual-layer redundancy retention for a period of time can be selected; while for temporary or high-frequency but non-critical data, asynchronous deletion can be triggered immediately to avoid wasting low-performance tier resources.

[0062] In this embodiment, when a data migration operation in a specified directory is triggered, the storage fragments within a file are classified and identified based on the read and write operation records of each storage fragment in the specified directory within a preset statistical period. For files with both read and write operations within the statistical period, the entire file is migrated to ensure data consistency. For files with only read operations within the statistical period, only the storage fragments that are actually read are migrated. This achieves fine-grained migration at the fragment level, thereby significantly reducing the amount of redundant data migrated to the high-performance storage layer, effectively controlling the space occupancy rate of high-cost storage media, and reducing the overall storage cost of the system. In addition, by determining the data to be migrated in the above manner, the bandwidth utilization efficiency and response speed of the migration operation are improved while preserving the access performance of hot data, avoiding excessive consumption of system I / O resources by copying too many large files in full.

[0063] In some embodiments, to accurately identify the access frequency and operation patterns of files in a specified directory and improve the efficiency of operations to determine data to be migrated, a frequency statistics set can be maintained for the specified directory based on read or write operations received by the distributed file system within the time interval (i.e., the statistical period) from the start of the statistical event to the end of the statistical event. This set is used for real-time monitoring and analysis of file access behavior. The frequency statistics set uses access records as storage units, and each access record includes at least a file identifier and an operation type. The operation type characterizes the access characteristics of the file pointed to by the file identifier within the current statistical period and can include at least the following three types of markers: read marker, write marker, and file migration marker.

[0064] In any access record, the read flag indicates that none of the storage segments corresponding to the file identifier in the access record have been written to, and at least one storage segment has been read to, which means that the file pointed to by the file identifier is currently only being read and not being written to.

[0065] A write flag in any access record indicates that none of the storage segments corresponding to the file identifier in that access record have been read, and at least one storage segment has been written. In other words, it means that the file pointed to by that file identifier is currently only being written and not being read.

[0066] A file migration marker in any access record indicates that at least one storage fragment corresponding to the file identifier in that access record has undergone a write operation and at least one storage fragment has undergone a read operation. If an access record includes a file migration marker, it means that all storage fragments corresponding to the file identifier in that access record satisfy the first condition.

[0067] See Figure 3AThe illustrated diagram shows the structure of a popularity statistics set. Each access record in this set includes at least two fields: file identifier and operation type. The file identifier field value represents the unique identifier of the file segment accessed by a read or write operation received within the current statistical period for any file under the specified directory. The operation type is dynamically adjusted based on the read or write operations received within the current statistical period. If the operation type in any access record is marked as read, when a write operation is received for the storage segment corresponding to the file identifier in that access record, the read operation is updated to a file migration marker; otherwise, the read marker remains unchanged. If the operation type in any access record is marked as write, when a read operation is received for the storage segment corresponding to the file identifier in that access record, the read operation is updated to a file migration marker; otherwise, the write marker remains unchanged. If the operation type in any access record is marked as file migration, when a read or write operation is received for the storage segment corresponding to the file identifier in that access record, the file migration marker remains unchanged.

[0068] Through the three types of operation markers mentioned above, the distributed file system can dynamically construct and update the popularity statistics set based on the received read and write operations within the current statistical period, thereby accurately distinguishing files with different access modes. Based on this, when the statistical end event is detected, all storage shards of files with file migration markers in the popularity statistics set will be migrated as a whole, only the storage shards of files with read markers will be migrated, and files with only write markers will not be migrated for the time being.

[0069] Based on this, the detection in the foregoing embodiments that all storage fragments corresponding to any file identifier in the specified directory satisfy the first condition may include the existence of an access record corresponding to that file identifier in the popularity statistics set, and the operation type in that access record being a file migration marker when the data migration operation in the specified directory is triggered. Correspondingly, the detection that all storage fragments corresponding to any file identifier in the specified directory satisfy the second condition may include the existence of an access record corresponding to that file identifier in the popularity statistics set, and the operation type in that access record being a read marker when the data migration operation in the specified directory is triggered.

[0070] Each access record within this popularity statistics set is generated based on read or write operations received by the distributed file system within the statistical period for any file in the specified directory. See also Figure 3BThe illustrative flowchart illustrates a step for updating the heat statistics set when a read operation for any file in a specified directory is received within the current statistical period. After detecting the start event of the statistics corresponding to the specified directory and before the end event of the statistics, i.e., within the statistical period before the data migration operation is triggered, the heat statistics set is updated through the following operations when a read operation for any file in the specified directory is received: S301a, query whether there is a first access record in the heat statistics set that matches the file identifier carried by the read operation; S302a, if it exists, then if the operation type in the first access record is a write flag, update the write flag to the file migration flag; S303a, if it does not exist, a new access record is created in the heat statistics set; the file identifier in the new access record is the file identifier carried by the read operation, and the operation type is read tag.

[0071] In other words, the system dynamically maintains the popularity statistics set within the current statistical period to ensure that each access record accurately reflects the latest access pattern of the corresponding file. When a read operation is received, the system first attempts to check in the popularity statistics set whether an access record for the file to which the storage shard accessed by the read operation belongs already exists. If it already exists and its operation type is marked as write (indicating that only write operations have occurred previously), it means that the file is now exhibiting mixed "read + write" behavior, so its operation type is changed to file migration mark. Correspondingly, if no record for the file exists, it means that the read operation is the first access to the file within this statistical period and is a read operation, so a new access record with the operation type marked as read is created for the file.

[0072] Accordingly, see Figure 3C The illustrative flowchart illustrates the steps for updating the heat statistics set when a write operation to any file in a specified directory is received within the current statistical period. In the statistical period prior to the data migration operation being triggered, upon receiving a write operation to any file in the specified directory, the heat statistics set is updated through the following operations: S301b, query whether there is a second access record in the heat statistics set that matches the file identifier carried by the write operation; S302b, if it exists, then if the operation type in the second access record is a read marker, update the read marker to the file migration marker; S303b, if it does not exist, then create a new access record in the heat statistics set; the file identifier in the new access record is the file identifier carried by the write operation, and the operation type is write tag.

[0073] In other words, the system's processing logic for write operations is symmetrical to that for read operations within the statistical period, ensuring that each access record accurately reflects the latest access pattern of the corresponding file. When a write operation is received, the system first attempts to check in the popularity statistics set whether an access record for the file to which the storage shard accessed by the write operation belongs already exists. If it already exists and its operation type is marked as read (indicating that only read operations have occurred previously), it means that the file is now exhibiting mixed "read + write" behavior, so its operation type is changed to file migration mark. Correspondingly, if no record for the file exists, it means that the write operation is the first access to the file within this statistical period and is a write operation, so a new access record with the operation type marked as write is created for the file.

[0074] For example, suppose the statistical period for a specified directory has just begun (i.e., the statistical start event has been triggered), and the popularity statistics set is initially empty: (1) Time point T1: A write operation to file File_A in the execution directory is received; The query for the popularity statistics set did not find a record for File_A. Create a new access record R1: {File ID: File_A, Operation Type: Write Flag}.

[0075] (2) Time point T2: A read operation on file File_B is received.

[0076] The query for the popularity statistics set did not find a record for File_B. Create a new access record R2: {File ID: File_B, Operation Type: Read Marker}.

[0077] (3) Time point T3: A read operation on File_A is received.

[0078] Query the popularity statistics set and find the matching record R1 {File_A, write flag}; Since the current operation is a read operation, and the original operation type was a write flag, the operation type of this record is updated to: {File ID: File_A, Operation Type: File Migration Flag}.

[0079] (4) Time point T4: Another read operation on File_B is received.

[0080] Query the popularity statistics set and find the matching record R2{File_B, read flag}; Since the operation type is still a read operation, and the original operation type was a read flag, no changes are needed.

[0081] If the data migration operation for the specified directory is triggered at the end of the statistical period T5, then: Obtain access records R1 containing file migration markers from the popularity statistics set; determine that file identifier File_A points to a file whose storage shard meets the first condition; and use the data carried by all storage shards corresponding to file identifier File_A as the data to be migrated; and, Obtain access records R2 that include read tags from the heat statistics set, determine the file identifier File_B that points to the file in the storage shard that meets the second condition, and take the data carried by the storage shard that only performs read operations among all the storage shards corresponding to the file identifier File_B as the data to be migrated.

[0082] The above methods ensure that the heat statistics set can capture the evolution of file access patterns in real time and accurately during the current data migration cycle, thus providing a reliable data basis for determining the data to be migrated.

[0083] In some embodiments, each access record in the popularity statistics set may further include a set of read storage fragments; the set of read storage fragments is used to record the identifiers of the storage fragments that have undergone read operations corresponding to the file identifiers in the access record. Based on this, during the statistical period when the data migration operation is triggered, if a read operation is received for any file in the specified directory, the identifier of each storage fragment read by the read operation is stored in the set of read storage fragments in the first access record or the new access record. That is, if a first access record exists in the popularity statistics set that matches the file identifier carried by the read operation, the identifier of each storage fragment read by the read operation is stored in the set of read storage fragments in the first access record; otherwise, after a new access record is created, the identifier of each storage fragment read by the read operation is stored in the set of read storage fragments in the newly created access record.

[0084] Furthermore, during the process of storing the identifier of each storage fragment read by the read operation into the set of read storage fragments, for the storage fragment to be stored in the set, if the set of read storage fragments already includes the identifier of the storage fragment, the read operation count corresponding to the storage fragment is updated; if the set of read storage fragments does not include the identifier of the storage fragment, the identifier of the storage fragment is recorded in the set of read storage fragments and the read operation count is initialized.

[0085] In other words, within the current statistical period, a set of read storage fragments can be maintained to record which specific storage fragments have been actually read and to count the number of read operations. This number of read operations reflects the access frequency of each storage fragment, enabling finer-grained migration of storage fragments that only perform read operations. When processing each read operation, one or more storage fragments involved in the operation are parsed, and the following operations are performed on each read storage fragment: if the identifier of the storage fragment already exists in the set of read storage fragments, its corresponding read operation count is incremented by 1 to reflect its access frequency; if the identifier of the storage fragment has not yet been recorded in the set of read storage fragments, the identifier of the storage fragment is added to the set of read storage fragments, and the read operation count is initialized to 1. Based on this, the relevant read information of each storage fragment read by the read operation is stored in the set of read storage fragments, including recording the fragment identifier and the number of read operations for that storage fragment.

[0086] Based on the read operation count of the read storage shards introduced in the above embodiments, a read operation count threshold mechanism can be introduced to distinguish between high-frequency hotspot reads and occasional low-frequency reads, thereby avoiding the mistaken migration of cold shards accessed by chance to the high-performance layer. Based on this, when the data carried by the storage shard corresponding to the file identifier that has only been read is taken as the data to be migrated, the access record matching the file identifier can be queried from the hotspot statistics set first, and the read operation count corresponding to each storage shard in the read storage shard set can be obtained from the access record; furthermore, each storage shard in the read storage shard set whose read operation count exceeds the set threshold is taken as the data to be migrated.

[0087] For example, suppose file File_Y is only read during the current statistical period, and its access record in the popularity statistics set is {File ID: File_Y, Operation Type: Read Flag}. This file contains four storage shards: S1, S2, S3, and S4. The maintained read storage shard set records are as follows: {S1, Read Operation Count: 4}, {S2, Read Operation Count: 1}, {S4, Read Operation Count: 2}. If the set read operation count threshold is 2, then the data carried by S1 and S4 will be considered as data to be migrated and moved to the high-performance layer storage area when the statistics end event is triggered.

[0088] By introducing a shard-level read hotness threshold filtering method, and under the condition that all storage shards corresponding to the file identifier meet the second condition, fine-grained control of high-frequency hot reads and occasional low-frequency reads is further realized, effectively balancing performance improvement and storage cost, and avoiding resource waste caused by low-value data migration in the high-performance layer.

[0089] To enable those skilled in the art to better understand the data storage management method provided in this application, this embodiment illustrates the data storage management method by maintaining a popularity statistics set in a specified directory. See [link to relevant documentation]. Figure 4A An exemplary flowchart of another data storage management method is shown, which can be implemented in the following manner: S401: When any specified directory under the distributed file system is configured with a one-time or periodic migration policy based on access time, a set of heat statistics is assigned to that specified directory. S402, based on the read or write operations received for any file in the specified directory within the statistical period, update the popularity statistics set; Updating a popularity statistics set includes updating existing access records in the set or adding new access records. For example... Figure 4B This example illustrates a schematic diagram of a single I / O operation processing within a statistical period, for any received I / O operation: (1) This I / O operation is a write operation. S4021a, Query whether there is an access record in the popularity statistics set that matches the file identifier carried by the write operation; That is, check whether there is an access record in the existing access records of the heat statistics set that contains the same file identifier as the file identifier carried by the write operation.

[0090] S4022a, if it exists, check whether the operation type included in the matching access record is a read tag; S4023a, if it is a read flag, then update the read flag included in the matched access record to a file migration flag; otherwise, execute step S4024a to keep the operation type included in the matched access record unchanged.

[0091] S4025a, if no matching access record exists, a new access record is created and stored in the popularity statistics set. The file identifier of the new access record is the same as the file identifier carried by the write operation, and the operation type is write tag.

[0092] Since the access record includes a set of read storage fragments, and the file migration marker indicates that all data carried by all storage fragments of the file will be migrated, the set of read storage fragments included in the access record can be cleared to save memory space.

[0093] (2) This I / O operation is a read operation. S4021b, Query whether there is an access record in the popularity statistics set that matches the file identifier carried by the read operation; That is, check whether there is an access record in the existing access records of the heat statistics set that contains the same file identifier as the file identifier carried by the read operation.

[0094] S4022b, if it exists, check whether the operation type included in the matching access record is a write flag; S4023b, if it is a write flag, then update the write flag included in the matching access record to a file migration flag; otherwise, execute step S4024b to check whether the operation type included in the matching access record is a read flag, and if it is a read flag, execute step S4025b to record the fragment identifier of each storage fragment accessed by the read operation to the set of read storage fragments included in the matching access record, and update the number of read operations corresponding to the fragment identifier; Specifically, if the fragment identifier already exists in the set of read storage fragments included in the matching access record, the read operation count corresponding to the existing fragment identifier is incremented; otherwise, the read operation count corresponding to the fragment identifier is initialized to 1.

[0095] S4026b, if no matching access record exists, a new access record is created and stored in the popularity statistics set. The file identifier of the new access record is the same as the file identifier carried by the read operation, the operation type is read tag, and the read storage fragment set includes the fragment identifier and the number of read operations for each storage fragment accessed by the read operation.

[0096] The above method is used to process a single I / O operation within the statistical period. For each read or write operation received within the statistical period, the popularity statistics set is updated in the above manner.

[0097] S403, when the data migration operation of the specified directory is triggered at the end of the statistical period, traverse each access record in the popularity statistics set; S404, Check whether the operation type included in the access record is a read tag; S405, if it is a read marker (satisfying the second condition), then the data carried by the storage fragments recorded in the set of read storage fragments included in the access record is taken as the data to be migrated; otherwise, step S406 is executed to detect whether the operation type included in the access record is a file migration marker. S407 If it is a file migration marker (satisfying the first condition), then the data carried by all storage fragments corresponding to the file identifier included in the access record is taken as the data to be migrated; otherwise, it is determined that all storage fragments corresponding to the file identifier in the access record will not be migrated (step S408).

[0098] S409, after the heat statistics set has been traversed, the determined data to be migrated is migrated from the low-performance layer storage area to the high-performance layer storage area, and the heat statistics set is cleared.

[0099] When a periodic migration strategy is configured, a new statistical cycle will be triggered after the data to be migrated is determined. During this new statistical cycle, read or write operations on any file in the specified directory will be continuously monitored and recorded in the heat statistics set after it has been cleared.

[0100] As can be seen from the above, this embodiment, based on the characteristic that reads only require partial sharding and writes require full updates in EC / multi-replica scenarios, achieves on-demand migration in data migration scenarios from low-performance layers to high-performance layers, thereby optimizing the response performance of read and write operations. This method targets a designated directory for data migration from the low-performance layer to the high-performance layer, and determines the data to be migrated based on the storage sharding access characteristics of each file accessed in that directory within a statistical period.

[0101] For example, assuming the read flag is 0, the write flag is 11, and the file migration flag is 1, then: (1) If there are file-based read and write operations within the statistical period, the operation type (migration mark position) in the access record corresponding to the file will be set to 1, indicating that the migration of all storage fragments of all file segments of the entire file will be performed to ensure read and write performance.

[0102] (2) If there are only file-based read operations during the statistical period, the operation type (migration marker position) in the access record corresponding to the file is set to 0, and all read storage fragments are recorded. Subsequently, only read storage fragments are migrated to save high-performance storage space.

[0103] (3) If there are no read or write operations on the file during the statistical period, then no information of the file will be recorded in the popularity statistics set, and the file will not be migrated in the future.

[0104] (4) If there are only file-based write operations during the statistical period, the operation type (migration marker position) in the access record corresponding to the file is set to 11, and the file is not migrated.

[0105] Based on this, when a write operation arrives within the statistical period: If a file is not recorded in the popularity statistics set, then the file's iNode information (i.e., file identifier) ​​is recorded, and the migration marker position is 11; If the file is already recorded in the popularity statistics set, check if the migration flag is 0. If it is, update it to 1 and clear the read storage fragment set of the file. Otherwise, do not update (migration flag is 1 or 11).

[0106] When a read operation arrives within the statistical period: If a file is not recorded in the popularity statistics set, the file iNode information is recorded, the migration mark position is 0, the accessed storage shard is recorded in the read storage shard set, and its popularity value is set to 1.

[0107] If a file is already recorded in the popularity statistics set and the migration flag is 11, then the migration flag is updated to 1, and the read storage fragment set of the file is cleared. If the file is already recorded in the popularity statistics set and the migration flag is 0, determine whether the accessed storage fragment is already in the read storage fragment set. If so, increment the number of read operations on the accessed storage fragment; otherwise, write the accessed storage fragment to the read storage fragment set and set its popularity value to 1. Otherwise, no update is performed (migration flag is 1).

[0108] Based on the data storage management method described in the foregoing embodiments, this embodiment takes the data migration operation in the EC (4+2) scenario as an example for illustrative explanation.

[0109] Assuming directory dir3 uses an EC 4+2 redundancy strategy, with a fixed chunk size of 64 MB per file segment, each data shard is 64 / 4 = 16 MB. The user configures a migration task to occur every Monday at 22:00. The directory contains 100 files, each 2 GB in size. Therefore, each file is split into 2 GB ÷ 64 MB = 32 segments. Assuming the statistical period is within the following timeframes: file1 is accessed and appended once, and the migration marker position is 1; Files 2 through 10 are read-only, with the migration flag set to 0. The set of read-only storage fragments includes [chunk0:D0,D1,D2,D3, popularity 2; chunk1:D0,D1,D2,D3, popularity 2]. So, if the data migration operation is triggered at 22:00 on Monday: All storage fragments corresponding to the 32 file segments Chunk of file1 are migrated to the high-performance layer. The actual space occupied in the high-performance layer storage area is: 32×64 MB×(4+2) / 4 = 3GB; Files 2 through 10 (9 files in total, flag bit = 0) are each promoted to 2 hot chunks, with each chunk containing 4 fragments. The remaining file fragments remain in the low-performance layer. The actual space occupied in the high-performance layer storage area is: 9 × 2 × 16 × 4 MB = 1.125 GB. Additional storage space required for the high-performance layer: 3 GB + 1.125 GB = 4.125 GB In contrast, if a traditional full-file migration scheme (migrating all files from file1 to file10 to the high-performance layer) is used, the actual space occupied in the high-performance layer storage area would be 10 × 2 × 6 / 4 GB = 30 GB. The data storage management method proposed in this embodiment saves approximately 86% of the high-performance layer resources compared to the full-file migration schemes of related technologies, improves the utilization efficiency of high-cost storage media, and still ensures the access performance of frequently accessed data.

[0110] Corresponding to the embodiments of the aforementioned data storage management method, see [link to relevant documentation]. Figure 5 As shown, this application also provides an embodiment of a data storage management device applied to a distributed file system, wherein the data storage space within the distributed file system is divided into a high-performance layer storage area and a low-performance layer storage area; the device includes: The migration data determination module 501 is configured to perform the following operations to determine the data to be migrated in the specified directory when a data migration operation in the specified directory is triggered in the distributed file system: If it is detected that all storage shards corresponding to any file identifier in the specified directory meet a first condition, then the data carried by all storage shards corresponding to that file identifier is taken as the data to be migrated; the first condition includes that at least one storage shard has been written to and at least one storage shard has been read to within a statistical period; the statistical period is a preset time window and its end time is aligned with the trigger time of the data migration operation; If it is detected that all storage shards corresponding to any file identifier in the specified directory meet a second condition, then the data carried by the storage shard that has only been read to within the storage shards corresponding to that file identifier is taken as the data to be migrated; the second condition includes that no write operation has been performed on any storage shards within the statistical period and at least one storage shard has been read to within the statistical period. The migration operation execution module 502 is configured to migrate the determined data to be migrated from the low-performance layer storage area to the high-performance layer storage area.

[0111] In some embodiments, a popularity statistics set is maintained for the specified directory; the popularity statistics set is stored in the form of access records; each access record includes at least a file identifier and an operation type; the operation type includes at least a read flag, a write flag, and a file migration flag; A read flag in any access record indicates that none of the storage segments corresponding to the file identifier in that access record have been written to, and at least one storage segment has been read to. A write flag in any access record indicates that none of the storage segments corresponding to the file identifier in that access record have been read, and at least one storage segment has been written. A file migration marker in any access record indicates that at least one storage segment corresponding to the file identifier in that access record has been written to and at least one storage segment has been read from.

[0112] In some embodiments, detecting that all storage fragments corresponding to any file identifier in the specified directory satisfy the first condition includes: The heat statistics set contains access records corresponding to the file identifier, and the operation type in the access record is file migration marker.

[0113] In some embodiments, detecting that all storage fragments corresponding to any file identifier in the specified directory satisfy the second condition includes: When the access record corresponding to the file identifier exists in the heat statistics set, and the operation type in the access record is a read marker when the statistics end event corresponding to the specified directory is detected.

[0114] In some embodiments, before the data migration operation of the specified directory is triggered, the apparatus further includes a read operation processing module configured to: If a read operation is received for any file in the specified directory within the statistical period, the system queries the popularity statistics set to see if there is a first access record that matches the file identifier carried by the read operation. If it exists, and the operation type in the first access record is a write flag, the write flag is updated to the file migration flag. If it does not exist, a new access record is created in the popularity statistics set. The file identifier in the new access record is the file identifier carried by the read operation, and the operation type is a read flag.

[0115] In some embodiments, each access record further includes a set of read storage fragments; the set of read storage fragments is used to record the identifiers of the storage fragments that have undergone read operations corresponding to the file identifiers in the access record; the apparatus further includes a read fragment statistics module, configured to: If a read operation is received for any file in the specified directory within the statistical period, the identifier of each storage segment read by the read operation is stored in the set of read storage segments in the first access record or the new access record; Specifically, during the process of storing the identifier of each storage segment read by the read operation into the set of read storage segments, if the set of read storage segments already includes the identifier of the storage segment, the read operation count corresponding to the storage segment is updated; if the set of read storage segments does not include the identifier of the storage segment, the identifier of the storage segment is recorded in the set of read storage segments and the read operation count is initialized.

[0116] In some embodiments, when the migration data determination module is configured to use data carried by storage fragments that have only undergone read operations among all storage fragments corresponding to the file identifier as the data to be migrated, it includes: The system queries the access records that match the file identifier from the heat statistics set, and obtains the number of read operations corresponding to each storage fragment in the read storage fragment set from the access records; each storage fragment in the read storage fragment set whose number of read operations exceeds a set threshold is regarded as data to be migrated.

[0117] In some embodiments, before the data migration operation of the specified directory is triggered, the apparatus further includes a write operation processing module configured to: If a write operation is received for any file in the specified directory within the statistical period, the system queries the popularity statistics set to see if there is a second access record that matches the file identifier carried by the write operation. If it exists, and the operation type in the second access record is a read marker, the read marker is updated to the file migration marker. If it does not exist, a new access record is created in the popularity statistics set. The file identifier in the new access record is the file identifier carried by the write operation, and the operation type is a write marker.

[0118] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0119] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0120] This application also provides an electronic device, the structural schematic diagram of which is shown below. Figure 6 As shown, the electronic device 600 includes at least one processor 601, a memory 602, and a bus 603. At least one processor 601 is electrically connected to the memory 602. The memory 602 is configured to store at least one computer-executable instruction, and the processor 601 is configured to execute the at least one computer-executable instruction to perform the steps of any data storage management method provided in any embodiment or optional implementation of this application.

[0121] Furthermore, the processor 601 can be an FPGA (Field-Programmable Gate Array) or other devices with logic processing capabilities, such as an MCU (Microcontroller Unit) or a CPU (Central Processing Unit).

[0122] This application also provides another readable storage medium storing a computer program that, when executed by a processor, implements the steps of any data storage management method provided in any embodiment or optional implementation of this application.

[0123] The readable storage media provided in this application include, but are not limited to, any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic cards, or optical cards. In other words, readable storage media include any medium by which a device (e.g., a computer) stores or transmits information in a readable form.

[0124] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0125] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A data storage management method characterized by comprising: Applied to a distributed file system, the data storage space within the distributed file system is divided into a high-performance layer storage area and a low-performance layer storage area; the method includes: When a data migration operation is triggered in a specified directory within the distributed file system, the following operations are performed to determine the data to be migrated in that specified directory: If all storage shards corresponding to any file identifier in the specified directory meet the first condition, then the data carried by all storage shards corresponding to that file identifier will be used as the data to be migrated. The first condition includes that at least one storage shard has been written to and at least one storage shard has been read to within the statistical period. The statistical period is a preset time window and its end time is aligned with the trigger time of the data migration operation. If it is detected that all storage fragments corresponding to any file identifier in the specified directory meet the second condition, then the data carried by the storage fragments that have only performed read operations among all storage fragments corresponding to that file identifier will be regarded as the data to be migrated; the second condition includes that no write operations have been performed on any storage fragments during the statistical period and at least one storage fragment has performed a read operation. The identified data to be migrated is moved from the low-performance layer storage area to the high-performance layer storage area.

2. The method of claim 1, wherein, A popularity statistics set is maintained for the specified directory; the popularity statistics set is stored in the form of access records; each access record includes at least a file identifier and an operation type; the operation type includes at least a read flag, a write flag, and a file migration flag; A read flag in any access record indicates that none of the storage segments corresponding to the file identifier in that access record have been written to, and at least one storage segment has been read to. A write flag in any access record indicates that none of the storage segments corresponding to the file identifier in that access record have been read, and at least one storage segment has been written. A file migration marker in any access record indicates that at least one storage segment corresponding to the file identifier in that access record has been written to and at least one storage segment has been read from.

3. The method according to claim 2, characterized in that, The detection that all storage fragments corresponding to any file identifier in the specified directory satisfy the first condition includes: The heat statistics set contains access records corresponding to the file identifier, and the operation type in the access record is file migration marker.

4. The method of claim 2, wherein, The condition that all storage fragments corresponding to any file identifier in the specified directory satisfy the second condition includes: When the access record corresponding to the file identifier exists in the heat statistics set, and the operation type in the access record is a read marker when the statistics end event corresponding to the specified directory is detected.

5. The method of claim 2, wherein, Before the data migration operation in the specified directory is triggered, the method further includes: If a read operation is received for any file in the specified directory within the statistical period, query whether there is a first access record in the popularity statistics set that matches the file identifier carried by the read operation. If it exists, then if the operation type in the first access record is a write flag, update the write flag to the file migration flag; If it does not exist, a new access record is created in the heat statistics set; the file identifier in the new access record is the file identifier carried by the read operation, and the operation type is read tag.

6. The method of claim 5, wherein, Each access record also includes a set of read storage fragments; this set of read storage fragments is used to record the identifiers of the storage fragments that have undergone read operations corresponding to the file identifier in the access record; the method further includes: If a read operation is received for any file in the specified directory within the statistical period, the identifier of each storage segment read by the read operation is stored in the set of read storage segments in the first access record or the new access record; Specifically, during the process of storing the identifier of each storage segment read by the read operation into the set of read storage segments, if the set of read storage segments already includes the identifier of the storage segment, the read operation count corresponding to the storage segment is updated; if the set of read storage segments does not include the identifier of the storage segment, the identifier of the storage segment is recorded in the set of read storage segments and the read operation count is initialized.

7. The method of claim 6, wherein, The data carried by the storage fragments that have only undergone read operations among all storage fragments corresponding to this file identifier are taken as the data to be migrated, including: Query the access records that match the file identifier from the heat statistics set, and obtain the number of read operations corresponding to each storage segment in the read storage segment set from the access records; Each storage fragment in the set of read storage fragments whose number of read operations exceeds a set threshold is considered as data to be migrated.

8. The method of claim 2, wherein, Before the data migration operation in the specified directory is triggered, the method further includes: If a write operation is received for any file in the specified directory within the statistical period, query whether there is a second access record in the popularity statistics set that matches the file identifier carried by the write operation. If it exists, then if the operation type in the second access record is a read marker, update the read marker to the file migration marker; If it does not exist, a new access record is created in the heat statistics set; the file identifier in the new access record is the file identifier carried by the write operation, and the operation type is write tag.

9. A data storage management apparatus, characterized by comprising: The device is applied to a distributed file system, wherein the data storage space within the distributed file system is divided into a high-performance layer storage area and a low-performance layer storage area; the device includes: The migration data determination module is configured to perform the following operations to determine the data to be migrated in the specified directory when a data migration operation in the specified directory is triggered in the distributed file system: If it is detected that all storage shards corresponding to any file identifier in the specified directory meet a first condition, then the data carried by all storage shards corresponding to that file identifier is taken as the data to be migrated; the first condition includes that at least one storage shard has been written to and at least one storage shard has been read to within a statistical period; the statistical period is a preset time window and its end time is aligned with the trigger time of the data migration operation; If it is detected that all storage shards corresponding to any file identifier in the specified directory meet a second condition, then the data carried by the storage shard that has only been read to within the storage shards corresponding to that file identifier is taken as the data to be migrated; the second condition includes that no write operation has been performed on any storage shards within the statistical period and at least one storage shard has been read to within the statistical period. The migration operation execution module is configured to migrate the determined data to be migrated from the low-performance layer storage area to the high-performance layer storage area.

10. An electronic device, comprising: include: Memory, processor; The memory is used to store computer programs; The processor is configured to invoke the computer program to implement the method as described in any one of claims 1-8.

11. A readable storage medium, having stored thereon a computer program, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-8.

Images