Converged storage methods and systems for cross-storage clusters
By building a converged storage system across storage clusters, a unified access view is provided, and multi-dimensional hot and cold data identification and automatic migration are performed. This solves the problems of unbalanced and wasteful storage resource management in existing technologies, and achieves performance and cost optimization and efficient resource utilization.
Patent Information
- Application Number
- CN202611612517.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-19
- Publication Date
- 2026-11-17
AI Technical Summary
Existing storage resource management solutions lack a unified storage view across clusters, resulting in uneven resource utilization among storage clusters, high storage costs, and resource waste caused by mixing data at different temperatures.
By building converged storage instances, a unified storage access view is provided. Based on access patterns, time characteristics, and business value, multi-dimensional hot and cold data identification is performed, and hot data is automatically migrated to high-performance storage clusters, while cold data is migrated to large-capacity storage clusters. A dual-track scanning mechanism and asynchronous scoring calculation are used to ensure the efficiency and accuracy of the identification and migration process.
It enables unified and transparent access across storage clusters, optimizes performance and cost, improves storage resource utilization, ensures the accuracy and consistency of data migration, and reduces storage management complexity.
Smart Images

Figure CN123114384A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of distributed storage technology, and in particular to a converged storage method and system for cross-storage clusters. Background Technology
[0002] With the rapid development of technologies such as big data, artificial intelligence, and high-performance computing, data storage demands are becoming increasingly diversified and complex. In typical computing tasks, the performance requirements for storage systems differ significantly at different stages: data preprocessing and model training stages require high throughput and latency performance, while stages such as result archiving and log storage prioritize storage capacity and unit storage cost. To meet these differentiated needs, data centers typically deploy multiple types of storage clusters simultaneously, such as storage clusters built with solid-state drives (SSDs) and those built with hard disk drives (HDDs), allowing users to choose according to their task requirements.
[0003] However, existing storage resource management solutions have the following problems: First, there is a lack of a unified storage view across clusters. In existing solutions, when a user submits a task, they must select one from multiple storage clusters and mount the selected storage instance to the task's data directory. All data generated during task execution is stored within that storage instance. Due to the lack of a unified namespace and access interface between different storage clusters, users cannot use multiple storage clusters simultaneously in a single task.
[0004] Second, uneven resource utilization among storage clusters leads to high storage costs. For complex tasks involving multiple stages, users often select storage clusters based on the highest performance requirements to meet the peak performance demands that may occur at each stage of the task. This results in a severe imbalance in load between different types of storage clusters, causing users to pay extra costs for a large amount of data that could be stored in low-cost storage throughout the entire task lifecycle.
[0005] Third, mixing data at different temperatures within a storage instance leads to a waste of storage resources. In a single storage instance, frequently accessed "hot" data from the early stages of a task is mixed with "cold" data from the later stages. Cold data occupies high-performance storage resources for a long time, while hot data may face migration or degradation due to insufficient storage space. Summary of the Invention
[0006] This application provides a converged storage method and system for cross-storage clusters. By constructing converged storage instances and automatically migrating multi-dimensional hot and cold data, it achieves unified and transparent access across heterogeneous storage clusters and automatic flow of hot data between high-performance storage and cold data between large-capacity storage, thereby balancing business performance requirements and storage cost control.
[0007] Firstly, this application provides a converged storage method for cross-storage clusters, including: Receive a user's request to create a converged storage instance, the request carrying the user's quota requirement information for different types of storage clusters; Based on the quota requirement information, corresponding types of storage resources are allocated from at least two heterogeneous storage clusters to construct a converged storage instance and provide users with a unified storage access view. The data within the converged storage instance is identified as hot or cold. The data is comprehensively scored from multiple dimensions based on access pattern characteristics, time characteristics, and business value characteristics. The data is then marked as hot or cold data according to the comprehensive scoring results. Based on the hot and cold data labeling results, hot data is automatically migrated to a high-performance storage cluster, and cold data is migrated to a large-capacity storage cluster.
[0008] Preferably, the method uses an adaptation layer to provide protocol adapters for different types of underlying storage clusters, and encapsulates the heterogeneous protocols of each underlying storage cluster into a standard file system interface. In response to a user performing a file read / write operation under the unified storage access view, the adaptation layer routes the read / write operation to the corresponding physical storage cluster based on the file's metadata information.
[0009] Preferably, a dual-track scanning mechanism is used to scan the data within the converged storage instance, the dual-track scanning mechanism including: Access-triggered scan: Activated immediately when data is accessed, updating the access record of the data and triggering a reassessment of the data's hot / cold status; Bypass audit trace scan: During periods when the load of the converged storage instance is below a preset threshold, the metadata is scanned in a sampling manner. The access-triggered scan and the bypass audit trail scan are both executed asynchronously, independent of the data read / write I / O path of the converged storage instance; the scan operation and the scoring calculation operation are subject to dual rate limiting of concurrency and I / O bandwidth.
[0010] Preferably, the multi-dimensional comprehensive score includes: Access pattern scoring is determined based on the timeliness and frequency of data access. Based on the creation and modification times of the data, a time decay strategy is used to determine the time characteristic score; Determine business value scores based on user-defined business tags; The access pattern score, the time characteristic score, and the business value score are weighted and fused to obtain a comprehensive score. The hot and cold labels of the data are determined based on the relationship between the comprehensive score and a preset threshold.
[0011] Preferably, the method further includes: receiving a user's custom hot / cold identifier for specified data within the converged storage instance via a hot / cold data tagging API interface; the custom hot / cold identifier has a higher priority than the automatic identification result of the hot / cold status identification.
[0012] Preferably, the automatic migration of hot data to a high-performance storage cluster and the migration of cold data to a large-capacity storage cluster includes: A migration plan is generated based on the latest hot and cold data labeling results. The migration plan includes: migrating newly added hot data from the large-capacity storage cluster to the high-performance storage cluster, and migrating newly added cold data from the high-performance storage cluster to the large-capacity storage cluster. The migration plan is executed in batches according to priority, with the amount of data migrated in each batch constrained by a preset concurrent migration limit.
[0013] Preferably, for each data file to be migrated, the following migration process is performed: Read data from the source storage cluster and calculate the first checksum; Transmit data to the target storage cluster; After the write operation to the target storage cluster is completed, a second checksum is calculated and compared with the first checksum. When the first checksum matches the second checksum, the metadata of the converged storage instance is updated, and the physical location of the data file is pointed to the target storage cluster.
[0014] Preferably, the migration process further includes an exception handling mechanism: When the first checksum is inconsistent with the second checksum, the migration operation is automatically retried; If the migration process is interrupted, the offset of the transmitted data is recorded through the breakpoint resume mechanism, and the transmission continues from the breakpoint after recovery; When the number of retries reaches the preset maximum number of retries, an automatic rollback is triggered, deleting the incomplete data already written to the target storage cluster and restoring the metadata to its state before migration.
[0015] Secondly, this application provides a converged storage system for cross-storage clusters, including: The user layer is used to receive requests from users to create converged storage instances, the requests carrying quota requirement information for different types of storage clusters specified by the user. The unified storage view module is used to allocate corresponding types of storage resources from at least two heterogeneous storage clusters according to the quota requirement information, construct a converged storage instance, and provide users with a unified storage access view. The hot and cold data identification module is used to identify the hot and cold status of data in the converged storage instance. It performs a multi-dimensional comprehensive score on the data based on access pattern characteristics, time characteristic characteristics, and business value characteristics, and marks the data as hot data or cold data according to the comprehensive score result. The automatic data migration module is used to automatically migrate hot data to a high-performance storage cluster and cold data to a large-capacity storage cluster based on the hot and cold data labeling results.
[0016] This application also provides an electronic device including one or more processors for implementing the methods described in any of the preceding claims or implementing the functions of the systems described above.
[0017] Compared with the prior art, the beneficial effects of this application include at least the following: By providing a unified storage view across storage clusters and an automatic hot and cold data separation mechanism, hot data is placed on high-performance storage to ensure business performance, while cold data is migrated to low-cost, high-capacity storage to reduce overhead, thus achieving an optimized balance between performance and cost and significantly improving overall storage utilization.
[0018] By intelligently identifying and automatically migrating hot and cold data, data at different temperatures is rationally distributed across different types of storage clusters, avoiding over-occupancy of high-performance storage clusters and underutilization of large-capacity storage clusters, thus achieving dynamic load balancing between clusters.
[0019] The system employs a comprehensive scoring strategy that considers access patterns, time characteristics, and business value, achieving a cold / hot identification accuracy rate of over 90%. The lightweight scanning mechanism has minimal impact on normal I / O (input / output) performance. Checksum, breakpoint resume, and automatic rollback mechanisms ensure the consistency and reliability of data migration, with automatic data transfer latency controlled within 30 minutes.
[0020] It provides users with a unified view of storage access, so users don't need to care about the actual physical storage location of the data. It also supports users to customize the hot and cold data identification through the API (Application Programming Interface), so as to retain the flexibility of personalized configuration while automating management. Attached Figure Description
[0021] Figure 1 The diagram shown is a schematic of a converged storage method for cross-storage clusters provided in an embodiment of this application; Figure 2 The diagram shown is a schematic of a converged storage process for cross-storage clusters provided in an embodiment of this application. Figure 3 The diagram shown is a schematic flowchart of the multi-dimensional hot and cold data judgment process provided in an embodiment of this application. Figure 4 The diagram shown is a schematic representation of the data migration process provided in an embodiment of this application. Figure 5 The diagram shown is a schematic of a converged storage system architecture for cross-storage clusters provided in an embodiment of this application. Detailed Implementation
[0022] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.
[0023] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.
[0024] Example 1, referring to Figure 1 , Figure 2 This embodiment provides a converged storage method for cross-storage clusters, including: Step 1: Receive a user's request to create a converged storage instance, the request carrying the user's quota requirement information for different types of storage clusters; Step 2: Based on the quota requirement information, allocate corresponding types of storage resources from at least two heterogeneous storage clusters to construct a converged storage instance and provide users with a unified storage access view; Step 3: Identify the hot and cold status of the data within the converged storage instance. Based on access pattern characteristics, time characteristics, and business value characteristics, perform a multi-dimensional comprehensive score on the data and mark the data as hot or cold data according to the comprehensive score results. Step 4: Based on the hot and cold data labeling results, automatically migrate hot data to a high-performance storage cluster and cold data to a large-capacity storage cluster.
[0025] In step 1, the quota requirement information includes high-performance storage cluster quotas and large-capacity storage cluster quotas; the method also includes: differentiated storage pricing based on the quota requirement information, whereby users only need to pay for the types of storage resources actually used.
[0026] The method uses an adaptation layer to provide protocol adapters for different types of underlying storage clusters, and encapsulates the heterogeneous protocols of each underlying storage cluster into a standard file system interface. In response to a user performing a file read / write operation under the unified storage access view, the adaptation layer routes the read / write operation to the corresponding physical storage cluster based on the file's metadata information.
[0027] The underlying storage cluster type includes any combination of NFS (Network File System) protocol clusters, CIFS (Common Internet File System) protocol clusters, S3 (Simple Storage Service) object storage clusters, and HDFS (Hadoop Distributed File System) distributed file system clusters; the standard file system interface is a POSIX (Portable Operating System Interface) compatible interface.
[0028] High-performance storage clusters refer to storage clusters composed of low-latency, high-IOPS (Input / Output Operations Per Second) storage media (such as NVMe SSDs, where NVMe stands for Non-Volatile Memory Express and SSD stands for Solid State Drive) used to provide high-speed data read and write services. Large-capacity storage clusters refer to storage clusters composed of low-cost, high-capacity storage media (such as HDDs, where HDDs are used) used to provide large-capacity data storage services. The specific storage media types mentioned above are merely examples and do not constitute a limitation on the scope of protection of this invention.
[0029] Specifically, users initiate a request to create a converged storage instance through a unified task submission platform or storage management interface. In this request, users need to specify quota requirements for different types of storage clusters. For example, a 100GB quota for a high-performance storage cluster is used to store hot data such as training datasets and model parameters; a 1TB quota for a large-capacity storage cluster is used to store cold data such as log files, intermediate checkpoints, and historical archives. This quota requirement information is used by the system for differentiated storage pricing, and users only need to pay for the storage resources they actually use. Through this quota specification and differentiated pricing method, users can flexibly configure resources according to the actual storage needs at each stage of the task, avoiding paying for unused storage capacity and effectively reducing storage costs.
[0030] Upon receiving a creation request, storage space is allocated from the pre-connected storage cluster pool, based on quota requirements, from both high-performance and high-capacity storage clusters. This storage space from different physical clusters is organized into a logical converged storage instance. A converged storage instance is a logical storage unit constructed by allocating storage resources from multiple heterogeneous storage clusters according to the user's quota requirements; it has an independent storage access path.
[0031] The adaptation layer implements protocol adapters for different protocol types of underlying storage clusters, uniformly encapsulating the heterogeneous protocols of each underlying storage cluster into a standard file system interface. The types of underlying storage clusters include any combination of NFS protocol clusters, CIFS protocol clusters, S3 object storage clusters, and HDFS distributed file system clusters, and the standard file system interface is a POSIX-compliant interface. Specifically, the S3 object storage cluster refers to an object storage system compatible with the AWS S3 API interface.
[0032] Under this unified view, users see a complete logical storage space without needing to be aware of the underlying physical distribution of data. When a user performs file read / write operations, the adaptation layer routes the read / write operation to the corresponding physical storage cluster based on the file's metadata information; the entire process is completely transparent to the user.
[0033] The system assesses the hot / cold status of data within converged storage instances. For each data file being evaluated, a comprehensive score is awarded based on three dimensions: access patterns, time characteristics, and business value. The score determines whether the data is considered "hot" or "cold." This mechanism identifies data at different "temperatures" within the storage instance, providing a basis for differentiated data storage decisions.
[0034] Based on the hot and cold data labeling results, hot data is automatically migrated to a high-performance storage cluster to ensure data access performance, while cold data is migrated to a large-capacity storage cluster to reduce storage costs, thus enabling automatic flow of data at different temperatures between different types of storage clusters.
[0035] Therefore, by constructing converged storage instances, a unified storage access view across heterogeneous storage clusters is provided to users. Users can complete all data read and write operations in a single logical storage space without worrying about the physical distribution of the underlying data. Through automatic hot and cold data identification, the system can sense the temperature characteristics of different data in the storage instance. Through automatic data migration, the system schedules the flow of data between different types of storage clusters based on the hot and cold data identification results, thereby balancing business performance requirements and storage cost control.
[0036] In one possible implementation, the step of allocating corresponding types of storage resources from at least two heterogeneous storage clusters based on the quota requirement information to construct a converged storage instance includes: The system retrieves quota requirement information for each type of storage cluster from user requests. This quota requirement information includes the required storage capacity for each type of storage cluster; for example, a 100GB quota for high-performance storage and a 1TB quota for large-capacity storage. The quota requirement information also includes a storage type identifier for each type of storage cluster, used to instruct the system to allocate resources to the corresponding type of storage cluster. Based on the quota requirement information, resource allocation requests are sent to the corresponding heterogeneous storage clusters respectively; Each of the heterogeneous storage clusters allocates a corresponding capacity of storage space from its storage resource pool according to the resource allocation request; The allocated storage spaces from different heterogeneous storage clusters are logically aggregated to generate a converged storage instance, and a unified metadata namespace is created for the converged storage instance. Specifically, the storage spaces returned by each storage cluster are incorporated into unified metadata management, an independent metadata namespace is created for the converged storage instance, and a mapping relationship between logical paths and physical storage locations is established, so that the storage spaces from different heterogeneous storage clusters are logically presented as a unified storage space.
[0037] The effects of the above technical solution are as follows: By acquiring quota requirement information and initiating resource allocation to various types of storage clusters, unified scheduling of on-demand storage resource allocation in heterogeneous storage clusters is achieved. Through logical aggregation and the creation of a unified metadata namespace, storage spaces from different physical clusters are integrated into a logically complete storage instance, enabling users to access cross-cluster storage resources under a unified namespace without needing to concern themselves with the actual physical location of the underlying data, thus reducing the complexity of storage management. Users only need to specify the quota requirements for each type of storage when creating the instance, and the system can automatically complete resource allocation and logical aggregation across heterogeneous clusters, improving the efficiency of storage resource delivery.
[0038] In one possible implementation, the step of allocating corresponding types of storage resources from at least two heterogeneous storage clusters based on the quota requirement information to construct a converged storage instance includes: The adapter layer establishes a connection with each underlying storage cluster, uses the corresponding protocol adapter to mount the allocated physical storage space to the unified resource pool, and registers the key attribute information of each storage resource with the metadata management module. The metadata management module creates a unified metadata namespace for the converged storage instance and maintains the mapping relationship between logical paths and physical storage locations. The system provides users with a unified storage access view. When a user initiates a file read / write request under this view, the adaptation layer routes the request to the corresponding physical storage cluster based on the mapping relationship.
[0039] Specifically, a converged storage instance is constructed in the following way: Resource Mounting and Adaptor Layer Registration: First, the adaptation layer establishes connections with each underlying storage cluster. For each storage cluster, the adaptation layer uses the corresponding protocol adapter to mount the allocated physical storage space to the unified resource pool of the converged storage system. After mounting, the adaptation layer registers the storage resource with the system's metadata management module and reports its key attribute information, including storage type, protocol type, storage capacity, and access address. The storage type includes high-performance and high-capacity types; the protocol adapters include NFS, CIFS, S3, and HDFS protocol adapters.
[0040] Building Global Metadata and Mapping Relationships: After receiving the registration information of all storage resources, the system's metadata management module creates an independent, globally unified metadata namespace for each newly created converged storage instance. Specifically, this includes: creating a root directory for the instance within the global namespace as the unified entry point for user access; and maintaining a mapping table that records the correspondence between logical paths and physical storage locations within the converged storage instance, including the physical storage cluster to which files under the logical path belong and their specific storage locations within that cluster.
[0041] Generate a unified logical storage view: After completing the above steps, the system presents a unified storage access view to the user; when the user accesses this view through the standard file system interface, they see a logical storage space with a unified directory structure, without needing to know the actual physical storage location of the data.
[0042] I / O request routing and forwarding: When a user initiates a file read / write request through the unified view, the adaptation layer intercepts the request, parses the logical path in the request, queries the mapping table, finds the physical storage cluster and its access address corresponding to the logical path; then, the adaptation layer calls the corresponding protocol adapter to convert the user's standardized I / O request into a native protocol request that the target storage cluster can recognize, and forwards it to the corresponding physical storage cluster for execution.
[0043] The effects of the above technical solution are as follows: By mounting resources and registering adaptation layers, heterogeneous storage clusters with different protocol types can be uniformly connected to the converged storage system, enabling cross-protocol storage resource management. By constructing global metadata and mapping relationships, an independent namespace and logical-to-physical mapping are established for each converged storage instance, logically integrating storage spaces from different physical clusters into a complete storage instance. By generating a unified logical storage view, users can access cross-cluster storage resources through a standard file system interface without being aware of the heterogeneity and physical distribution of the underlying storage. Through I / O request routing and forwarding mechanisms, every read and write operation by the user is automatically routed to the correct physical storage cluster, and the entire process is completely transparent to the user, achieving unified storage access across heterogeneous storage clusters.
[0044] Reference Figure 3 In one possible implementation, a dual-track scanning mechanism is used to scan the data within the converged storage instance. This dual-track scanning mechanism includes: Access-triggered scan: Activated immediately when data is accessed, updating the access record of the data and triggering a reassessment of the data's hot / cold status; Bypass audit trace scan: During periods when the load of the converged storage instance is below a preset threshold, the metadata is scanned in a sampling manner. The access-triggered scan and the bypass audit tracing scan are both executed asynchronously, independent of the data read / write I / O path of the converged storage instance. Dual rate limiting of concurrency and I / O bandwidth is applied to both scanning and scoring operations.
[0045] Specifically, when a user or application performs a read or write operation on a file in the converged storage instance, the access-triggered scanning mechanism is activated immediately, recording relevant information about the access, including metadata such as access timestamp, access type, and access source, and triggering a reassessment of the file's hot / cold status. Through this mechanism, the identification of hot data is real-time, ensuring that active data can be detected in a timely manner and maintained on the high-speed storage medium.
[0046] During periods of low load (such as early morning or when tasks are idle), a bypass audit trace scan is initiated to perform a full scan of the file metadata within the converged storage instance, identifying potentially cold data that has not been accessed for a long time. This mechanism ensures comprehensive cold data discovery, preventing cold data from consuming high-performance storage resources unnoticed for extended periods.
[0047] Both scanning and scoring operations are executed in independent threads or processes, without blocking normal user read and write requests, ensuring that real-time response to business needs is not affected.
[0048] Configure a rate limiting mechanism to restrict the number of concurrent threads for scanning and scoring operations, as well as the I / O bandwidth used during the scanning process, ensuring that the impact of the hot / cold status identification process on normal business operations is controlled within preset thresholds. For example, rate limiting parameters can be set to ensure that the increase in storage I / O latency does not exceed 5%, and the actual rate limiting threshold can be configured according to the specific deployment environment.
[0049] The beneficial effects of the above technical solution are as follows: Access-triggered scanning enables real-time reassessment of hot and cold data status when data is accessed, ensuring that hot data is detected in real time and kept on high-performance storage, thereby guaranteeing business access performance. Bypass audit tracing scanning enables comprehensive scanning of metadata during off-peak periods, promptly identifying long-unaccessed cold data and migrating it to low-cost storage, avoiding waste of storage resources. Through asynchronous execution mechanisms, scanning and scoring operations are independent of the normal I / O path, without affecting users' real-time read and write requests. Through dual rate limiting mechanisms, the concurrency and I / O bandwidth of scanning operations are constrained, ensuring that the performance impact of the hot and cold status identification process on normal business is always controlled within an acceptable range, thus ensuring stable operation while accurately identifying hot and cold data.
[0050] Reference Figure 3 In one possible implementation, the multi-dimensional comprehensive score includes: Access pattern scoring is determined based on the timeliness and frequency of data access. Based on the creation and modification times of the data, a time decay strategy is used to determine the time characteristic score; Determine business value scores based on user-defined business tags; The access pattern score, the time characteristic score, and the business value score are weighted and fused to obtain a comprehensive score. The hot and cold labels of the data are determined based on the relationship between the comprehensive score and a preset threshold.
[0051] The business value score is set by the user through the API interface for specific data, or a default value is used.
[0052] Specifically, the multi-dimensional comprehensive score includes the following three dimensions: Access pattern scoring: determined based on the timeliness and frequency of access data. It takes into account both the most recent access time (Recency) and the access frequency (Frequency).
[0053] Specifically, define the time decay function. ,in The time difference between the current time and the most recent access time. The decay coefficient is used to characterize the impact of data access timeliness on the score; the most recently accessed data receives a higher decay function value.
[0054] Define frequency normalization value ,in This represents the actual access frequency. This value represents the preset reference frequency and is used to characterize the impact of data access frequency on the score.
[0055] Access pattern rating ,in This is a timeliness weighting coefficient used to adjust the relative importance of access timeliness and access frequency in access pattern scoring, serving as a preferred option. .
[0056] Time characteristic score ( The time decay strategy is used to determine the time based on the creation and modification time of the data.
[0057] Specifically, defining freshness ,in The time difference since the data was created. The time difference between the last modification of the data and the present. This is the freshness decay coefficient. To modify the weighting coefficients; newly created or recently modified data receives a higher freshness value, which gradually decreases over time; typically This indicates that the most recently modified data has higher popularity.
[0058] Business value score ( ): Consider user-defined business tags. Users can tag specific data with business value labels through the system's API interface. The label value range is [0,1], where 1 represents the highest business value (must be kept as hot data), and 0 represents the lowest business value (can be archived as cold data). If the user does not define a custom tag, the default value is used. .
[0059] The scores from the three dimensions mentioned above are weighted and fused to obtain the comprehensive heat value. ,in Preset thermal data threshold and cold data threshold ( ):when When, it is marked as hot data; when When, mark as cold data; when In such cases, the current state remains unchanged (hysteresis mechanism to avoid frequent switching). Furthermore, if the user explicitly specifies a hot or cold indicator for certain data via the API interface, this custom indicator takes precedence over the automatic scoring result.
[0060] The hot data threshold and the cold data threshold are determined according to at least one of the following methods: Method 1: Perform statistical analysis on the comprehensive score of all data in the converged storage instance, and use the comprehensive score value corresponding to the preset first percentile (e.g., the 70th percentile) as the hot data threshold, and use the comprehensive score value corresponding to the preset second percentile (e.g., the 70th percentile) as the cold data threshold, wherein the first percentile is greater than the second percentile. Method 2: The hot data threshold is determined based on the ratio of the total capacity of the high-performance storage cluster to the total capacity of the converged storage instance; the cold data threshold is determined based on the ratio of the total capacity of the large-capacity storage cluster to the total capacity of the converged storage instance; the system assigns a comprehensive score to all data within the converged storage instance, from highest to lowest. The comprehensive score at the proportional position is used as the initial hot data threshold, and the data ranked after sorting is then processed. The overall score at the proportional position is used as the initial cold data threshold, where This refers to the proportion of high-performance storage cluster capacity to the total instance capacity. This refers to the proportion of the capacity of the large-capacity storage cluster to the total capacity of the instance.
[0061] Method 3: Dynamically adjust the hot data threshold and the cold data threshold based on the current used capacity ratio of the high-performance storage cluster. When the current used capacity ratio exceeds a first capacity threshold (e.g., 80%), increase the hot data threshold; when the current used capacity ratio is lower than a second capacity threshold (e.g., 40%), decrease the hot data threshold. Specifically, the adjustment amount of the hot data threshold is proportional to the difference between the current used capacity ratio of the high-performance storage cluster and the preset target capacity ratio; that is, the further the current used capacity ratio deviates from the target capacity ratio, the greater the adjustment of the hot data threshold. The cold data threshold is adjusted synchronously to maintain a constant hysteresis interval width between the hot data threshold and the cold data threshold.
[0062] The values of the aforementioned hot data threshold and cold data threshold must meet the following constraints: the hot data threshold is greater than the cold data threshold, and the difference between the hot data threshold and the cold data threshold is greater than or equal to the preset minimum hysteresis interval width. When the hot data threshold and cold data threshold calculated by any of the above methods do not meet the constraints, the system corrects the hot data threshold and cold data threshold based on the constraints to ensure that the hysteresis interval always has sufficient width to avoid frequent switching of data between hot and cold states.
[0063] The effects of the above technical solution are as follows: By incorporating a time decay function and frequency normalization value into the access pattern scoring, the system can accurately reflect the timeliness and frequency characteristics of data access, giving higher popularity scores to recently accessed or frequently accessed data. Similarly, by introducing a freshness function and setting a modification weight coefficient into the time characteristic scoring, newly created or recently modified data receives higher popularity scores, with modification operations having a greater weight on popularity than creation time alone. The business value scoring allows users to define custom business tags, enabling the system to determine data popularity based on business semantics. Finally, by weighting and fusing scores across these three dimensions, the system comprehensively reflects the data's access patterns and time characteristics. The overall popularity across three dimensions—business value, and overall popularity—combined with a multi-dimensional scoring mechanism improves the accuracy of hot and cold data identification. By setting hot and cold data thresholds and introducing a hysteresis mechanism, frequent switching between hot and cold data states is effectively avoided, ensuring the stability of data storage location. Allowing users to customize hot and cold data identifiers and set their priority above the automatic scoring results meets the personalized needs of businesses. Through asynchronous execution, dual rate limiting, and metadata caching mechanisms, the impact of the hot and cold status identification process on normal business is always kept within an acceptable range, while reducing the system overhead of repeated queries and improving the overall performance of the system.
[0064] In one possible implementation, the method further includes: receiving a user's custom hot / cold identifier for specified data within the converged storage instance via a hot / cold data tagging API interface; the custom hot / cold identifier has a higher priority than the automatic identification result of the hot / cold status identification.
[0065] Specifically, the system provides a hot / cold data labeling API interface, which allows users to customize hot / cold labels for specified data within the converged storage instance. This API interface allows users to specify the data file path or data identifier to be labeled, and to specify whether the data should be labeled as hot or cold data. For example, users can use the API call to force a model parameter file in a certain directory to be labeled as hot data, or to force a historical log file in that directory to be labeled as cold data.
[0066] In one possible implementation, when a user calls this API, they need to submit the following information: the identifier of the converged storage instance, the logical path of the data to be tagged within the converged storage instance, and a custom hot / cold identifier value (hot data or cold data). After receiving and parsing the request, the system writes the custom hot / cold identifier into the metadata of the data to be tagged.
[0067] In one possible implementation, when performing hot / cold status identification, for data with custom hot / cold identifiers set via an API interface, the custom identifier is preferentially used as the hot / cold labeling result, instead of using the automatic identification result of the multi-dimensional comprehensive scoring. Specifically, in the hot / cold status evaluation process for each data file, the system first checks whether a custom hot / cold identifier exists in the metadata of the data file; if it exists, the custom hot / cold identifier is directly used as the hot / cold labeling result for that data, and the multi-dimensional comprehensive scoring is not performed; if it does not exist, the multi-dimensional comprehensive scoring process is executed, and the hot / cold label is determined based on the scoring result.
[0068] In one possible implementation, users can also cancel the set custom hot / cold labels through this API interface. After cancellation, the hot / cold label of the data will be restored to the one determined by the system's automatic identification result.
[0069] Application scenario example: In AI training tasks, users can use an API to force the training dataset to be marked as "hot data," ensuring that the dataset remains on a high-performance storage cluster throughout the training process and guaranteeing data retrieval efficiency. Conversely, historical model checkpoints can be forcibly marked as "cold data" to avoid consuming high-performance storage space. This custom marking mechanism allows users to flexibly intervene in the hot / cold data identification results based on business semantics, meeting personalized needs.
[0070] The advantages of this implementation are as follows: By providing a hot and cold data tagging API interface, users can customize hot and cold tags for specific data according to business needs, which can flexibly adapt to the personalized needs of different businesses; by setting the priority of custom hot and cold tags to be higher than the automatic identification results, users' control over the storage location of specific data is ensured, avoiding the loss of access performance of critical business data or the waste of costs caused by archived data being stuck on high-performance storage media due to automatic identification errors; by supporting the cancellation of custom tags, users can restore the system's automatic identification and management at any time, ensuring the flexibility and reversibility of hot and cold data management.
[0071] Reference Figure 4 In one possible implementation, the automatic migration of hot data to a high-performance storage cluster and the migration of cold data to a large-capacity storage cluster includes: A migration plan is generated based on the latest hot and cold data labeling results. The migration plan includes: migrating newly added hot data from the large-capacity storage cluster to the high-performance storage cluster, and migrating newly added cold data from the high-performance storage cluster to the large-capacity storage cluster. The migration plan is executed in batches according to priority, with the amount of data migrated in each batch constrained by a preset concurrent migration limit.
[0072] In one possible implementation, the following migration process is performed for each data file to be migrated: Read data from the source storage cluster and calculate the first checksum; Transmit data to the target storage cluster; After the write operation to the target storage cluster is completed, a second checksum is calculated and compared with the first checksum. When the first checksum matches the second checksum, update the metadata of the converged storage instance and point the physical location of the data file to the target storage cluster; After the metadata update is successful, delete the original data copy from the source storage cluster.
[0073] The migration process also includes an exception handling mechanism: When the first checksum is inconsistent with the second checksum, the migration operation is automatically retried; If the migration process is interrupted, the offset of the transmitted data is recorded through the breakpoint resume mechanism, and the transmission continues from the breakpoint after recovery; When the number of retries reaches the preset maximum number of retries, an automatic rollback is triggered, deleting the incomplete data already written to the target storage cluster and restoring the metadata to its state before migration.
[0074] The effects of the above technical solution are as follows: Through a migration plan generation mechanism, the system only performs migration operations on data whose hot and cold data markings have changed, avoiding unnecessary data relocation and improving migration efficiency. By setting the migration priority of hot data to be higher than that of cold data, the system ensures that the access performance requirements of hot data are prioritized. By implementing batch execution and concurrency limits, the system avoids the impact on system bandwidth and storage I / O caused by a large number of migration tasks running simultaneously. By calculating the first checksum at the source end and the second checksum after the target end has written the data, and comparing them, the system ensures the integrity and consistency of data during the migration process, preventing data corruption due to transmission errors. By updating metadata before deletion... The order of source copies ensures that data can be recovered from the source even if anomalies occur during the migration process, guaranteeing data availability. Automatic retries for inconsistencies ensure that migration will ultimately succeed even if verification fails due to momentary errors. The breakpoint resume mechanism eliminates the need to retransmit all data after a break, allowing transmission to resume only from the point of interruption, significantly reducing network overhead and time costs associated with repeated transmissions. Automatic rollback is triggered when the retries reach their limit, deleting incomplete data on the target end and restoring metadata to its pre-migration state, ensuring data consistency and integrity in abnormal situations and avoiding issues such as dirty data residue or inconsistencies between metadata and physical location.
[0075] In one possible implementation, the step of prioritizing and executing the migration plan in batches includes: A migration urgency score is calculated for each data file to be migrated, and the data files are sorted in descending order of migration urgency score. The migration is then performed in batches according to the sorting results. The migration urgency score is determined based on at least one of the following dimensions: The difference between the overall score of the data file and a preset threshold is calculated according to the migration direction. For data to be migrated into high-performance storage, the difference is the difference between the overall score and the cold data threshold; for data to be migrated out of high-performance storage, the difference is the difference between the hot data threshold and the overall score. The rate of change of the comprehensive score of the data file is calculated based on the comprehensive score sequence of the data file over multiple evaluation periods, and the rate of change in the heating direction has a greater weight than the rate of change in the cooling direction. The size of the data file is assigned different weights based on the migration direction. When migrating into high-performance storage, large files have a higher weight than small files, while when migrating out of high-performance storage, small files have a higher weight than large files.
[0076] In one possible implementation, the migration urgency score is further determined based on at least one of the following dimensions: The acceleration of change in the overall score of the data file, wherein the acceleration of change is the amount of change in the rate of change between adjacent evaluation periods; The priority weight of the task or user to which the data file belongs; The current remaining capacity of the high-performance storage cluster is used to dynamically increase the urgency score of migration tasks involving data being moved out of the high-performance storage cluster when the current remaining capacity is lower than a preset urgency threshold.
[0077] In one possible implementation, the migration urgency score is a weighted sum of contribution values from multiple dimensions.
[0078] Specifically, the step of prioritizing and executing the migration plan in batches includes: A migration urgency score is calculated for each data file to be migrated, and the data files are sorted in descending order of migration urgency score. Migration is then performed in batches according to the sorting result. The migration urgency score is determined based on at least one of the following dimensions: The overall score offset dimension is determined based on the difference between the overall score of the data file and a preset threshold, and this difference is calculated separately according to the migration direction. For data to be migrated into high-performance storage (i.e., newly added hot data), the offset is the difference between the overall score and the cold data threshold. The larger the difference, the higher the popularity of the data, and the more priority it should be given to migrating into high-performance storage. For data to be migrated out of high-performance storage (i.e., newly added cold data), the offset is the difference between the hot data threshold and the overall score. The larger the difference, the higher the coldness of the data, and the more priority it should be given to migrating out of high-performance storage to free up space. By calculating the offset separately according to the migration direction, the urgency assessment for both hot data migration in and cold data migration out scenarios is more targeted.
[0079] Optionally, the above offset is normalized by dividing the offset by the difference between the hot data threshold and the cold data threshold, and mapping the offset to the [0,1] interval, so that the offsets between data with different comprehensive score distribution ranges are comparable.
[0080] Optionally, a boundary sensitivity coefficient is introduced to weight the normalized offset. The value range of the boundary sensitivity coefficient is... ,in The value is a constant greater than 1 (e.g., 1.5). When the normalized offset is less than or equal to a first preset threshold (e.g., 0.05), the boundary sensitivity coefficient reaches its maximum value; when the normalized offset is greater than a second preset threshold (e.g., 0.3), the boundary sensitivity coefficient reaches its minimum value of 1.0; when the normalized offset is between the first and second preset thresholds, the boundary sensitivity coefficient decreases as the offset increases (e.g., linearly). Through this boundary sensitivity coefficient, when the data is exactly on or very close to the threshold boundary, the offset contribution is amplified to the maximum extent; when the data is far from the threshold boundary, the amplification factor gradually decreases until it drops to 1.0, and no further amplification is performed, thereby preventing data from remaining unprocessed due to prolonged exposure to hot and cold boundaries.
[0081] The rate of change of the overall score is determined based on the rate of change of the overall score of the data file. This rate of change is calculated based on the overall score sequence of the data file over multiple evaluation periods. Specifically, the system records the historical overall score values of each data file over the most recent N evaluation periods. Let the overall score for the current evaluation period be... The overall score for the previous evaluation period was If the evaluation period is T, then the rate of change .
[0082] V>0 indicates that the data is heating up, and V<0 indicates that the data is cooling down. Furthermore, different weights are assigned based on the direction of change, with the rate of change in the heating direction having a higher weight than that in the cooling direction. This means that the urgency score for heating data increases more significantly than that for cooling data, reflecting an overall migration strategy that prioritizes hot data. For example, the weight for the heating direction is 1.2, and the weight for the cooling direction is 0.8.
[0083] Optionally, the rate of change also considers trends over a longer time span, using methods such as linear regression or exponential smoothing to calculate the rate of change based on the historical comprehensive score values of the most recent multiple periods, in order to eliminate the impact of a single abnormal fluctuation on the rate of change.
[0084] Data file size dimension: determined based on the size of the data files, with different weights assigned to each size according to the migration direction. For data to be migrated to high-performance storage, large files are migrated first because they incur greater performance loss on slow storage, and the performance gains after migration to high-performance storage are more significant. The weighting coefficient is positively correlated with the data size. For data to be migrated out of high-performance storage, small files are migrated first because they take less time to migrate, have less impact on system input / output (i.e., I / O), and can release storage space in a shorter time. The weighting coefficient is negatively correlated with the data size.
[0085] In one possible implementation, the migration urgency score is further determined based on at least one of the following dimensions: Overall score change acceleration dimension: the overall score change acceleration of the data file, where the change acceleration is the amount of change in the rate of change between adjacent evaluation periods; The rate of change is calculated based on the comprehensive score sequence of the data file over multiple evaluation periods, and different directional weights are assigned according to the direction of change, with the rate of change in the heating direction having a higher weight than the rate of change in the cooling direction. Specifically, acceleration change ,in, The rate of change for the current assessment period; The rate of change is T, representing the rate of change in the previous evaluation period, and the evaluation period duration is T. The acceleration of change reflects the momentum of changes in data popularity. When A<0 and When the data is cooling down (i.e., the cooling rate is accelerating), it indicates that the data is about to cool down rapidly, and the urgency to move it out of high-performance storage should be increased. A>0 and When the data is heating up (i.e., the data is heating up at an accelerating rate), it indicates that the data is about to become hot quickly, and the urgency to migrate it to high-performance storage should be increased. A>0 and When the data is cooling down, but the rate of cooling is slowing down, it indicates that the data may be about to stop falling and rebound, which can reduce the urgency of migrating out. When A<0 and When the data is heating up but the rate of increase is slowing down, it indicates that the data may be about to peak and fall back, which can reduce the urgency of migration.
[0086] Task or User Priority Dimension: Determined based on the priority weight of the task or user to which the data file belongs. The system configures different priority levels for different tasks or users. Data files of critical tasks or high-priority users receive a higher weight coefficient when calculating migration urgency, ensuring that the migration needs of critical business operations are prioritized. For example, the data migration urgency weight is 2.0 for real-time training tasks, 1.0 for offline analysis tasks, and 0.5 for archiving tasks.
[0087] Storage space urgency dimension: determined based on the current remaining capacity of the high-performance storage cluster. The system monitors the remaining capacity of the high-performance storage cluster in real time. When the current remaining capacity is lower than a preset urgency threshold, the urgency score of migration tasks involving data being moved out of the high-performance storage cluster is dynamically increased. For example, when the remaining capacity of the high-performance storage cluster is less than 10% of the total capacity, the system automatically multiplies the urgency score of cold data migration tasks by a space urgency coefficient (e.g., 1.5) to accelerate the migration of cold data to free up storage space and ensure that the high-performance storage cluster has sufficient available capacity to handle new hot data writes.
[0088] The effects of the above technical solution are as follows: By differentiating migration directions and calculating the comprehensive score offset separately, the urgency assessment for both hot data migration and cold data migration scenarios becomes more targeted. Introducing a boundary sensitivity coefficient gives extra priority to data approaching the threshold boundary, preventing data from remaining unprocessed for extended periods on the hot / cold boundary. By introducing a rate of change dimension and assigning different directional weights based on the direction of warming / cooling, the priority ranking reflects not only current popularity but also future trends. Introducing an acceleration dimension allows for the identification of accelerating cooling or warming rates, enabling adjustments to migration priorities before substantial changes in data popularity occur. This makes migration scheduling predictable; by weighting data files according to migration direction based on file size, large files are prioritized for migration into high-performance storage to maximize performance gains, while small files are prioritized for migration out of high-performance storage to quickly release storage space, achieving a balance between performance optimization and space release; by introducing task or user priority dimensions, the migration needs of critical business data are ensured to be prioritized; and by using storage space urgency dimensions, the urgency of cold data migration is dynamically increased when the capacity of the high-performance storage cluster is insufficient, enabling migration scheduling to adaptively respond to the capacity status of the storage system and avoid performance bottlenecks caused by the inability to migrate hot data due to the exhaustion of high-performance storage space.
[0089] In one possible implementation, the method includes: establishing a memory cache for frequently accessed metadata in hot / cold state identification.
[0090] During the hot / cold status identification process, the access frequency of metadata is statistically analyzed to identify frequently accessed metadata, which is then stored in a memory cache. This metadata includes the file's access timestamp, access frequency, creation time, modification time, business value score, hot / cold labeling results, and physical storage location information.
[0091] In one possible implementation, when querying metadata during each hot / cold state identification process, the memory cache is accessed first; if the target metadata exists in the memory cache, it is read directly from the memory cache without accessing the underlying metadata storage; if the target metadata does not exist in the memory cache, it is read from the underlying metadata storage and the reading result is stored in the memory cache.
[0092] In one possible implementation, the memory cache is managed using a preset eviction policy, such as the Least Recently Used (LRU) policy. When cache space is insufficient, metadata entries that have not been accessed for the longest time are evicted first. Evicted metadata entries will be reloaded into the memory cache when accessed subsequently.
[0093] In one possible implementation, an expiration period is set for metadata in the memory cache; when the metadata's lifetime in the cache exceeds the expiration period, the metadata entry is marked as invalid, and the system reloads the latest data from the underlying metadata storage the next time it accesses the metadata, so as to ensure the consistency between the cached data and the underlying stored data.
[0094] In one possible implementation, when the cold / hot tagging result or physical storage location of a data file changes, the corresponding metadata entry in the memory cache is synchronously updated to maintain consistency between the cached data and the underlying stored data. Specifically, when the cold / hot tagging of a data file changes or data migration is completed and metadata is updated, the system synchronously writes the updated metadata into the memory cache, replacing the original cache entry.
[0095] The beneficial effects of this implementation method are: By caching frequently accessed metadata in memory, metadata query operations during the hot / cold status identification process can directly retrieve results from the memory cache without frequent access to the underlying metadata storage, significantly reducing I / O overhead and latency for metadata queries and improving the overall efficiency of the hot / cold status identification process. By adopting cache eviction strategies such as LRU, the most frequently accessed metadata can be prioritized and retained within limited memory space, improving cache space utilization and hit rate. By setting cache expiration periods and change synchronization mechanisms, the consistency between cached data and underlying storage data is ensured, avoiding hot / cold status identification deviations or migration decision errors caused by untimely cache updates.
[0096] The following example illustrates the practical effects of this invention through a specific application scenario.
[0097] A user of an AI training platform submitted a deep learning model training task, which includes the following stages: During the data preprocessing stage, it is necessary to frequently and randomly read large-scale training datasets, which places high demands on storage I / O performance. During the model training phase, high-throughput writing of model checkpoints and training logs is required; The model evaluation and archiving phases after training have low storage performance requirements, but it is necessary to save the evaluation results and training logs.
[0098] Under the existing solution, in order to meet the performance requirements of the data preprocessing and training phases, users must mount the entire task's data directory to a high-performance storage cluster. After training is completed, a large amount of checkpoint files, historical logs, and other data still occupy high-performance storage space, resulting in resource waste and high costs.
[0099] After adopting the converged storage solution of this invention: Users create a converged storage instance, configuring a high-performance storage quota of 200GB and a large-capacity storage quota of 2TB. During data preprocessing and training phases, the training dataset is automatically identified as hot data and placed on high-performance storage to ensure training efficiency. Checkpoint files generated during training are initially marked as hot data. As training progresses and early checkpoints are no longer accessed, the system gradually downgrades them to cold data through multi-dimensional scoring and automatically migrates them to large-capacity storage. Evaluation results and log archives after training are also automatically classified as cold data and migrated to large-capacity storage. Ultimately, only currently actively used hot data is retained on high-performance storage, while historical archived data is stored on large-capacity storage, significantly reducing storage costs while ensuring training performance is not affected.
[0100] Example 2, refer to Figure 5 This embodiment provides a converged storage system for cross-storage clusters, used to implement the method described in Embodiment 1. The system includes: The user layer provides a task submission platform or storage management interface for receiving user requests to create converged storage instances. These requests carry user-specified quota requirements for different types of storage clusters. The user layer also provides a unified standard file system interface for users to access data within the converged storage system.
[0101] The converged storage subsystem comprises an access layer, a core processing layer, an adaptation layer, and a data management layer, wherein: The access layer includes a unified storage view module, a converged instance lifecycle management module, a global namespace mapping table module, and a quota management and differentiated pricing module.
[0102] The unified storage view module allocates corresponding types of storage resources from at least two heterogeneous storage clusters based on the quota requirement information, constructs a converged storage instance, and provides a unified storage access view to users through an adaptation layer. The quota requirement information includes quotas for high-performance storage clusters and quotas for large-capacity storage clusters. The quota management and differentiated pricing module performs differentiated storage pricing based on the quota requirement information, allowing users to pay only for the types of storage resources actually used. The converged instance lifecycle management module manages the lifecycle operations of converged storage instances, including creation, modification, and destruction. The global namespace mapping table module maintains the mapping relationship between logical paths and physical storage locations.
[0103] The core processing layer includes a hot / cold data identification module and a data scanning module.
[0104] The hot / cold data identification module is used to identify the hot / cold status of data within the converged storage instance. It performs a multi-dimensional comprehensive score on the data based on access pattern characteristics, time characteristics, and business value characteristics, and marks the data as hot or cold data according to the comprehensive score result. The hot / cold data identification module includes an access statistics engine, a time decay calculator, a score fusion decision-maker, and a user tagging manager. The access statistics engine is used to collect data on access frequency and access timeliness. The time decay calculator is used to calculate time decay based on the data's creation and modification times. The score fusion decision-maker is used to perform weighted fusion of scores from multiple dimensions and decide on the hot / cold label. The user tagging manager is used to manage custom hot / cold labels set by users for specified data through the hot / cold data tagging API interface.
[0105] The data scanning module scans the data within the converged storage instance using a dual-track scanning mechanism, including an access-triggered scanner and a bypass audit tracker. The access-triggered scanner activates immediately upon data access, updating the data's access record and triggering a reassessment of the data's hot / cold status. The bypass audit tracker performs a sampling-based traversal scan of metadata during periods when the converged storage instance's load is below a preset threshold. Both the access-triggered scanner and the bypass audit tracker execute asynchronously, independent of the converged storage instance's data read / write I / O paths. Dual rate limiting of concurrency and I / O bandwidth is applied to both scanning and scoring calculation operations.
[0106] The adaptation layer includes a storage adaptation module, which adopts a plug-in structure and contains multiple protocol adapters and a unified compatibility interface. Each protocol adapter adapts to different types of underlying storage clusters, encapsulating the heterogeneous protocols of each underlying storage cluster into a standard file system interface. The unified compatibility interface provides a consistent access interface to the upper layer. In response to a user performing file read / write operations under the unified storage access view, the adaptation layer routes the read / write operation to the corresponding physical storage cluster based on the file's metadata information.
[0107] The data management layer includes an automatic data migration module, which automatically migrates hot data to a high-performance storage cluster and cold data to a large-capacity storage cluster based on the hot and cold data labeling results. The automatic data migration module includes a migration plan generator, a data transfer engine, and an exception handling unit.
[0108] The migration plan generator is used to generate a migration plan based on the latest hot and cold data labeling results. The migration plan includes: migrating newly added hot data from the large-capacity storage cluster to the high-performance storage cluster, and migrating newly added cold data from the high-performance storage cluster to the large-capacity storage cluster; executing the migration plan in batches according to priority, with the amount of data migrated in each batch constrained by a preset concurrent migration limit.
[0109] The data transfer engine is used to perform the following migration process for each data file to be migrated: A migration plan is generated based on the latest hot and cold data labeling results. The migration plan includes: migrating newly added hot data from the large-capacity storage cluster to the high-performance storage cluster, and migrating newly added cold data from the high-performance storage cluster to the large-capacity storage cluster. The migration plan is executed in batches according to priority, with the amount of data migrated in each batch constrained by a preset concurrent migration limit.
[0110] For each data file to be migrated, perform the following migration process: Read data from the source storage cluster and calculate the first checksum; Transmit data to the target storage cluster; After the write operation to the target storage cluster is completed, a second checksum is calculated and compared with the first checksum. When the first checksum matches the second checksum, update the metadata of the converged storage instance and point the physical location of the data file to the target storage cluster; After the metadata update is successful, delete the original data copy from the source storage cluster.
[0111] The exception handling unit is used to: automatically retry the migration operation when the first checksum is inconsistent with the second checksum; when the migration process is interrupted, record the data offset that has been transmitted through the breakpoint resume mechanism, and continue the transmission from the breakpoint after recovery; when the number of retries reaches the preset retry limit, trigger automatic rollback, delete the incomplete data written in the target storage cluster, and restore the metadata to the state before migration.
[0112] In one possible implementation, the system further includes a hot and cold data tagging API interface for receiving custom hot and cold identifiers for specified data within the converged storage instance from a user; the custom hot and cold identifiers have a higher priority than the automatic identification results of the hot and cold data identification module.
[0113] In one possible implementation, the multi-dimensional comprehensive score includes: Access pattern scoring is determined based on the timeliness and frequency of data access. Based on the creation and modification times of the data, a time decay strategy is used to determine the time characteristic score; Determine business value scores based on user-defined business tags; The access pattern score, the time characteristic score, and the business value score are weighted and fused to obtain a comprehensive score. The hot and cold labels of the data are determined based on the relationship between the comprehensive score and a preset threshold.
[0114] The system also includes a metadata caching module, which is used to establish a memory cache for frequently accessed metadata during hot / cold status identification.
[0115] The specific implementation and beneficial effects of the above technical solution are the same as those in Embodiment 1, and will not be repeated here.
[0116] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method in embodiment 1 of this application or the functions of the system as described in embodiment 2 of this application.
[0117] This application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-readable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented using any method or technology. Information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media include, but are not limited to: phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.
[0118] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.
[0119] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, an element qualified by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
Claims
1. A converged storage method for cross-storage clusters, characterized in that, include: Receive a user's request to create a converged storage instance, the request carrying the user's quota requirement information for different types of storage clusters; Based on the quota requirement information, corresponding types of storage resources are allocated from at least two heterogeneous storage clusters to construct a converged storage instance and provide users with a unified storage access view. The data within the converged storage instance is identified as hot or cold. The data is comprehensively scored from multiple dimensions based on access pattern characteristics, time characteristics, and business value characteristics. The data is then marked as hot or cold data according to the comprehensive scoring results. Based on the hot and cold data labeling results, hot data is automatically migrated to a high-performance storage cluster, and cold data is migrated to a large-capacity storage cluster.
2. The method as described in claim 1, characterized in that, The method uses an adaptation layer to provide protocol adapters for different types of underlying storage clusters, and encapsulates the heterogeneous protocols of each underlying storage cluster into a standard file system interface. In response to a user performing a file read / write operation under the unified storage access view, the adaptation layer routes the read / write operation to the corresponding physical storage cluster based on the file's metadata information.
3. The method as described in claim 1, characterized in that, A dual-track scanning mechanism is used to scan the data within the converged storage instance. The dual-track scanning mechanism includes: Access-triggered scan: Activated immediately when data is accessed, updating the access record of the data and triggering a reassessment of the data's hot / cold status; Bypass audit trace scan: During periods when the load of the converged storage instance is below a preset threshold, the metadata is scanned in a sampling manner. The access-triggered scan and the bypass audit trail scan are both executed asynchronously, independent of the data read / write I / O path of the converged storage instance; the scan operation and the scoring calculation operation are subject to dual rate limiting of concurrency and I / O bandwidth.
4. The method according to claim 1, characterized in that, The multi-dimensional comprehensive score includes: Access pattern scoring is determined based on the timeliness and frequency of data access. Based on the creation and modification times of the data, a time decay strategy is used to determine the time characteristic score; Determine business value scores based on user-defined business tags; The access pattern score, the time characteristic score, and the business value score are weighted and fused to obtain a comprehensive score. The hot and cold labels of the data are determined based on the relationship between the comprehensive score and a preset threshold.
5. The method according to claim 1, characterized in that, The method further includes: receiving a user's custom hot / cold identifier for specified data within the converged storage instance via a hot / cold data tagging API interface; the custom hot / cold identifier has a higher priority than the automatic identification result of the hot / cold status identification.
6. The method according to claim 1, characterized in that, The automatic migration of hot data to a high-performance storage cluster and cold data to a large-capacity storage cluster includes: A migration plan is generated based on the latest hot and cold data labeling results. The migration plan includes: migrating newly added hot data from the large-capacity storage cluster to the high-performance storage cluster, and migrating newly added cold data from the high-performance storage cluster to the large-capacity storage cluster. The migration plan is executed in batches according to priority, with the amount of data migrated in each batch constrained by a preset concurrent migration limit.
7. The method according to claim 6, characterized in that, For each data file to be migrated, perform the following migration process: Read data from the source storage cluster and calculate the first checksum; Transmit data to the target storage cluster; After the write operation to the target storage cluster is completed, a second checksum is calculated and compared with the first checksum. When the first checksum matches the second checksum, the metadata of the converged storage instance is updated, and the physical location of the data file is pointed to the target storage cluster.
8. The method according to claim 7, characterized in that, The migration process also includes an exception handling mechanism: When the first checksum is inconsistent with the second checksum, the migration operation is automatically retried; If the migration process is interrupted, the offset of the transmitted data is recorded through the breakpoint resume mechanism, and the transmission continues from the breakpoint after recovery; When the number of retries reaches the preset maximum number of retries, an automatic rollback is triggered, deleting the incomplete data already written to the target storage cluster and restoring the metadata to its state before migration.
9. A converged storage system for cross-storage clusters, characterized in that, include: The user layer is used to receive requests from users to create converged storage instances, the requests carrying quota requirement information for different types of storage clusters specified by the user. The unified storage view module is used to allocate corresponding types of storage resources from at least two heterogeneous storage clusters according to the quota requirement information, construct a converged storage instance, and provide users with a unified storage access view. The hot and cold data identification module is used to identify the hot and cold status of data in the converged storage instance. It performs a multi-dimensional comprehensive score on the data based on access pattern characteristics, time characteristic characteristics, and business value characteristics, and marks the data as hot data or cold data according to the comprehensive score result. The automatic data migration module is used to automatically migrate hot data to a high-performance storage cluster and cold data to a large-capacity storage cluster based on the hot and cold data labeling results.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 8, or implements the function of the system according to claim 9.
