A hierarchical storage data method and apparatus

Abstract

The embodiment of the present application provides a kind of hierarchical storage data method and device, it is related to data processing technical field, the method is specifically as follows: receiving target data migration instruction for target file;In response to the data migration instruction for target file, first target data is migrated to high-performance storage end low-performance storage end and is saved, and / or second target data is migrated to the high-performance storage end and is saved;After the migration of first target data and / or second target data is completed, the change information corresponding to high-performance storage end and / or low-performance storage end is generated;The change information corresponding to high-performance storage end and low-performance storage end is obtained, and high-performance storage end and low-performance storage end are synchronized based on data change record.The present application can improve the efficiency when data is hierarchically stored, and improve data synchronization.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, specifically to a hierarchical data storage method and apparatus. Background Technology

[0002] In today's storage technology field, with the explosive growth of data volume and the ever-increasing demands for data processing performance, storage systems face numerous challenges. To meet diverse business needs, various storage technologies have emerged, among which tiered storage has become a prominent application.

[0003] High-performance storage systems such as parallel file systems play a crucial role in scenarios with extremely high read / write speed requirements due to their superior data access capabilities. For example, in large-scale scientific computing and high-performance data analysis, parallel file systems can rapidly process massive amounts of data, meeting the real-time data processing needs of researchers and enterprises. Meanwhile, low-performance systems, such as heterogeneous storage, like S3 object storage, are also widely used in data storage due to their significant advantages of low cost and high scalability, as well as the compatibility and versatility of NAS file protocols such as NFS. For instance, for enterprises with large amounts of data and high cost sensitivity, S3 object storage can provide cost-effective storage solutions; the NFS file protocol facilitates data sharing between different operating systems.

[0004] To optimize the allocation of storage resources and fully leverage the advantages of different storage systems, data flow technology has emerged. Through data flow, frequently accessed hot data can be stored in a higher-performance parallel file system to ensure fast data read and write speeds; while infrequently accessed cold data can be migrated to lower-cost heterogeneous storage. This effectively reduces storage costs while meeting business performance requirements, achieving an optimal combination of performance and cost.

[0005] However, currently, data import tasks only import metadata information, not the actual data. This means that when upper-layer applications on high-performance systems access data, they are highly likely to access files containing only metadata, triggering data flow I / O. This not only blocks application operations and severely impacts system performance, but also compromises data availability and reliability when the data flow system fails. Furthermore, when configuring bidirectional intelligent data flow, it's impossible to precisely ensure data consistency for bidirectional data flow over the same file, potentially leading to data corruption in practical applications and failing to meet enterprises' stringent requirements for data accuracy and integrity. Summary of the Invention

[0006] In view of this, embodiments of this application provide a hierarchical data storage method and apparatus, which can improve the efficiency of hierarchical data storage and improve data synchronization.

[0007] In a first aspect, embodiments of this application provide a hierarchical data storage method, applied in a target hierarchical storage management system in Lustre, including:

[0008] The system receives a target data migration instruction for a target file; the target data migration instruction is used to transfer first target data in the target file from a high-performance storage end to a low-performance storage end for storage, and / or transfer second target data in the target file from a low-performance storage end to a high-performance storage end for storage; the first target data includes first data and metadata corresponding to the first data; the second target data includes second data and metadata corresponding to the second data;

[0009] In response to the target data migration instruction, the first target data is migrated to the low-performance storage terminal for storage, and / or the second target data is migrated to the high-performance storage terminal for storage;

[0010] After the migration of the first target data and / or the second target data is completed, change information corresponding to the high-performance storage end and / or the low-performance storage end is generated;

[0011] Based on the change information, the high-performance storage terminal and the low-performance storage terminal will synchronize data; the change information includes the data change status and data change record status corresponding to the high-performance storage terminal or the low-performance storage terminal.

[0012] As an optional implementation of this application, before receiving the target data migration instruction for the target file, the method further includes:

[0013] Monitor the access frequency of the target data and obtain the access frequency of the target data;

[0014] When the target data is stored in the high-performance storage device and the access frequency is less than a first preset threshold; or when the target data is stored in the low-performance storage device and the access frequency is greater than a second preset threshold, the target data migration instruction is generated.

[0015] As an optional implementation of this application, the step of generating change information corresponding to the high-performance storage end and / or the low-performance storage end after the migration of the first target data and / or the second target data is completed includes:

[0016] The system periodically reads the change information corresponding to the high-performance storage terminal and the low-performance storage terminal from the database, and synchronizes the data between the high-performance storage terminal and the low-performance storage terminal based on the change information.

[0017] As an optional implementation of this application, the step of synchronizing data between the high-performance storage terminal and the low-performance storage terminal based on the change information includes:

[0018] Data synchronization is performed based on the data change status and data change record status corresponding to the high-performance storage terminal and the low-performance storage terminal, respectively.

[0019] As an optional implementation of this application, the data synchronization based on the data change status and data change record status corresponding to the high-performance storage terminal and the low-performance storage terminal respectively includes:

[0020] Determine whether the data change status of both the high-performance storage terminal and the low-performance storage terminal is "no data change" and whether the data change record status is "no data change record".

[0021] If so, then data synchronization is not required;

[0022] If not, data synchronization is performed based on the data change status and data change record status corresponding to the high-performance storage terminal and the low-performance storage terminal, respectively.

[0023] As an optional implementation of this application, the data synchronization based on the data change status and data change record status corresponding to the high-performance storage terminal and the low-performance storage terminal respectively includes:

[0024] If the data change status of the high-performance storage terminal or the low-performance storage terminal is "data changed", and the data change records of the high-performance storage terminal and the low-performance storage terminal are both "no data change records", then a log replay instruction is generated; the log replay instruction is used to instruct the replay of the first log residual records corresponding to the high-performance storage terminal and the low-performance storage terminal respectively, so as to perform data synchronization based on the first log residual records;

[0025] If the data change status of both the high-performance storage terminal and the low-performance storage terminal is "no data change", and the data change record status of either the high-performance storage terminal or the low-performance storage terminal is "no data change record", then a first data synchronization instruction is generated; the first data synchronization instruction is used to instruct the high-performance storage terminal or the low-performance storage terminal to perform data synchronization based on the first dirty data.

[0026] If the data change status of the high-performance storage terminal or the low-performance storage terminal is "data changed", and the data change record status of the high-performance storage terminal or the low-performance storage terminal is "data changed record", then a second data synchronization instruction is generated; after the second data synchronization instruction is executed, a second log replay instruction is generated; the second data synchronization instruction is used to instruct the high-performance storage terminal or the low-performance storage terminal to perform data synchronization based on the second dirty data; the second log replay instruction is used to instruct the replay of the second log residual records corresponding to the high-performance storage terminal and the low-performance storage terminal respectively, so as to perform data synchronization based on the second log residual records;

[0027] If both the high-performance storage terminal and the low-performance storage terminal have data change statuses, and both the high-performance storage terminal and the low-performance storage terminal have no data change records, then a third data synchronization instruction is generated. The third data synchronization instruction is used to instruct the high-performance storage terminal and the low-performance storage terminal to perform data synchronization based on the log version information of the fourth dirty data corresponding to their respective third dirty data.

[0028] If the data change status of both the high-performance storage terminal and the low-performance storage terminal is "no data change", and the data change record status of both the high-performance storage terminal and the low-performance storage terminal is "data change record", then a third log replay instruction is generated. The third log replay instruction instructs the replay of log residual records based on the log version information of the third log residual record and the fourth log residual record corresponding to the high-performance storage terminal and the low-performance storage terminal, respectively, so as to perform data synchronization based on the log version information.

[0029] If both the high-performance storage terminal and the low-performance storage terminal have data change statuses, and the data change record status of either the high-performance storage terminal or the low-performance storage terminal is "data change record", then a fourth data synchronization instruction is generated. After the fourth data synchronization instruction is executed, a fourth log synchronization instruction is generated. The fourth data synchronization instruction is used to instruct the high-performance storage terminal and the low-performance storage terminal to perform data synchronization based on the log version information of the sixth dirty data of their respective fifth dirty data. The fourth log replay instruction is used to instruct the replay of the fifth log residual records corresponding to the high-performance storage terminal and the low-performance storage terminal, so as to perform data synchronization based on the fifth log residual records.

[0030] If the data change status of either the high-performance storage terminal or the low-performance storage terminal has a data change status, and the data change record status of both the high-performance storage terminal and the low-performance storage terminal has a data change record, then a fifth data synchronization instruction is generated. After the fifth data synchronization instruction is executed, a fifth log synchronization instruction is generated. The fifth data synchronization instruction is used to instruct the high-performance storage terminal or the low-performance storage terminal to perform data synchronization based on the seventh dirty data. The fifth log replay instruction instructs the replay of log residual records based on the log version information of the sixth log residual record and the seventh log residual record corresponding to the high-performance storage terminal and the low-performance storage terminal, respectively, so as to perform data synchronization based on the log version information.

[0031] If both the high-performance storage terminal and the low-performance storage terminal have data change statuses, and both have data change record statuses, a sixth data synchronization instruction is generated. After the sixth data synchronization instruction is executed, a sixth log synchronization instruction is generated. The sixth data synchronization instruction is used to instruct the high-performance storage terminal and the low-performance storage terminal to synchronize data according to the log version information of their respective eighth dirty data and ninth dirty data. The sixth log replay instruction instructs the replay of log residual records according to the log version information of the eighth log residual records and the ninth log residual records corresponding to the high-performance storage terminal and the low-performance storage terminal, respectively, so as to synchronize data according to the log version information.

[0032] As an optional implementation of this application, before receiving the target data migration instruction for the target file, the method further includes:

[0033] The Consul component is integrated into the native Lustre tiered storage management system to obtain the target tiered storage management system.

[0034] Secondly, embodiments of this application provide a hierarchical data storage device, comprising:

[0035] A receiving unit is configured to receive a target data migration instruction for a target file; the target data migration instruction is configured to transfer first target data in the target file from a high-performance storage end to a low-performance storage end for storage, and / or transfer second target data in the target file from a low-performance storage end to a high-performance storage end for storage; the first target data includes first data and metadata corresponding to the first data; the second target data includes second data and metadata corresponding to the second data;

[0036] A migration unit is configured to, in response to the target data migration instruction, migrate the first target data to the low-performance storage terminal for storage, and / or migrate the second target data to the high-performance storage terminal for storage;

[0037] The generation unit is used to generate change information corresponding to the high-performance storage end and / or the low-performance storage end after the migration of the first target data and / or the second target data is completed.

[0038] A synchronization unit is used to synchronize data between the high-performance storage terminal and the low-performance storage terminal based on the change information; the change information includes the data change status and data change record status corresponding to the high-performance storage terminal or the low-performance storage terminal.

[0039] As an optional implementation of this application, the generation unit is specifically used to monitor the access frequency of the target data and obtain the access frequency of the target data; when the target data is stored in the high-performance storage terminal and the access frequency is less than a first preset threshold; or when the target data is stored in the low-performance storage terminal and the access frequency is greater than a second preset threshold, then the target data migration instruction is generated.

[0040] As an optional implementation of this application, the migration unit is further configured to periodically read the change information corresponding to the high-performance storage terminal and the low-performance storage terminal from the database, and synchronize the data of the high-performance storage terminal and the low-performance storage terminal based on the change information.

[0041] As an optional implementation of this application, the change information includes the data change status and data change record status corresponding to the high-performance storage terminal or the low-performance storage terminal;

[0042] The migration unit is also used to perform data synchronization based on the data change status corresponding to the high-performance storage end and the low-performance storage end, and the data change record status, respectively.

[0043] As an optional implementation of this application, the synchronization unit is specifically used to determine whether the data change status of the high-performance storage terminal and the low-performance storage terminal is both "no data change" and whether the data change record status is both "no data change record". If yes, then no data synchronization is required; if no, then data synchronization is performed based on the data change status and the data change record status corresponding to the high-performance storage terminal and the low-performance storage terminal, respectively.

[0044] As an optional implementation of this application, the synchronization unit is specifically used to generate a log replay instruction if the data change status of the high-performance storage terminal or the low-performance storage terminal is "data changed" and the data change records of the high-performance storage terminal and the low-performance storage terminal are both "no data change" records; the log replay instruction is used to instruct the replay of the first log residual records corresponding to the high-performance storage terminal and the low-performance storage terminal respectively, so as to perform data synchronization based on the first log residual records;

[0045] If the data change status of both the high-performance storage terminal and the low-performance storage terminal is "no data change", and the data change record status of either the high-performance storage terminal or the low-performance storage terminal is "no data change record", then a first data synchronization instruction is generated; the first data synchronization instruction is used to instruct the high-performance storage terminal or the low-performance storage terminal to perform data synchronization based on the first dirty data.

[0046] If the data change status of the high-performance storage terminal or the low-performance storage terminal is "data changed", and the data change record status of the high-performance storage terminal or the low-performance storage terminal is "data changed record", then a second data synchronization instruction is generated; after the second data synchronization instruction is executed, a second log replay instruction is generated; the second data synchronization instruction is used to instruct the high-performance storage terminal or the low-performance storage terminal to perform data synchronization based on the second dirty data; the second log replay instruction is used to instruct the replay of the second log residual records corresponding to the high-performance storage terminal and the low-performance storage terminal respectively, so as to perform data synchronization based on the second log residual records;

[0047] If both the high-performance storage terminal and the low-performance storage terminal have data change statuses, and both the high-performance storage terminal and the low-performance storage terminal have no data change records, then a third data synchronization instruction is generated. The third data synchronization instruction is used to instruct the high-performance storage terminal and the low-performance storage terminal to perform data synchronization based on the log version information of the fourth dirty data corresponding to their respective third dirty data.

[0048] If the data change status of both the high-performance storage terminal and the low-performance storage terminal is "no data change", and the data change record status of both the high-performance storage terminal and the low-performance storage terminal is "data change record", then a third log replay instruction is generated. The third log replay instruction instructs the replay of log residual records based on the log version information of the third log residual record and the fourth log residual record corresponding to the high-performance storage terminal and the low-performance storage terminal, respectively, so as to perform data synchronization based on the log version information.

[0049] If both the high-performance storage terminal and the low-performance storage terminal have data change statuses, and the data change record status of either the high-performance storage terminal or the low-performance storage terminal is "data change record", then a fourth data synchronization instruction is generated. After the fourth data synchronization instruction is executed, a fourth log synchronization instruction is generated. The fourth data synchronization instruction is used to instruct the high-performance storage terminal and the low-performance storage terminal to perform data synchronization based on the log version information of the sixth dirty data of their respective fifth dirty data. The fourth log replay instruction is used to instruct the replay of the fifth log residual records corresponding to the high-performance storage terminal and the low-performance storage terminal, so as to perform data synchronization based on the fifth log residual records.

[0050] If the data change status of either the high-performance storage terminal or the low-performance storage terminal has a data change status, and the data change record status of both the high-performance storage terminal and the low-performance storage terminal has a data change record, then a fifth data synchronization instruction is generated. After the fifth data synchronization instruction is executed, a fifth log synchronization instruction is generated. The fifth data synchronization instruction is used to instruct the high-performance storage terminal or the low-performance storage terminal to perform data synchronization based on the seventh dirty data. The fifth log replay instruction instructs the replay of log residual records based on the log version information of the sixth log residual record and the seventh log residual record corresponding to the high-performance storage terminal and the low-performance storage terminal, respectively, so as to perform data synchronization based on the log version information.

[0051] If both the high-performance storage terminal and the low-performance storage terminal have data change statuses, and both have data change record statuses, a sixth data synchronization instruction is generated. After the sixth data synchronization instruction is executed, a sixth log synchronization instruction is generated. The sixth data synchronization instruction is used to instruct the high-performance storage terminal and the low-performance storage terminal to synchronize data according to the log version information of their respective eighth dirty data and ninth dirty data. The sixth log replay instruction instructs the replay of log residual records according to the log version information of the eighth log residual records and the ninth log residual records corresponding to the high-performance storage terminal and the low-performance storage terminal, respectively, so as to synchronize data according to the log version information.

[0052] As an optional implementation of this application, the generation unit is further configured to mount the Consul component into the native Lustre hierarchical storage management system to obtain the target hierarchical storage management system.

[0053] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to cause the electronic device to implement the hierarchical data storage method described in any of the above embodiments when executing the computer program.

[0054] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a computing device, causes the computing device to implement the hierarchical data storage method described in any of the above embodiments.

[0055] The hierarchical data storage method provided in this application allows for data migration. Since the target data includes both the data itself and its corresponding metadata, the receiving end can directly process and analyze the data after migration, avoiding the problem of data being unusable due to only migrating metadata. Furthermore, since corresponding change information is generated after data migration, different strategies are employed during data synchronization based on the data change status and record status in the change information to synchronize data between high-performance and low-performance storage ends. This prevents data loss or overwriting, further improving system performance and stability. Attached Figure Description

[0056] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0057] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings that need to be called in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is one of the flowcharts of the hierarchical data storage method provided in the embodiments of this application;

[0059] Figure 2 This is an overall framework diagram of the hierarchical data storage method provided in the embodiments of this application;

[0060] Figure 3 The second flowchart of the hierarchical data storage method provided in the embodiments of this application;

[0061] Figure 4 This is one of the module interaction diagrams of the hierarchical data storage method provided in the embodiments of this application;

[0062] Figure 5The second module interaction diagram of the hierarchical data storage method provided in the embodiments of this application;

[0063] Figure 6 This is a schematic diagram of the structure of the hierarchical data storage device provided in the embodiments of this application;

[0064] Figure 7 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0065] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0066] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0067] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner. Furthermore, in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0068] It should be noted that, in this document, the term "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0069] This application provides a hierarchical data storage method, referring to... Figure 1 As shown, the hierarchical data storage method includes the following steps S101-S104:

[0070] S101, Receive the target data migration instruction for the target file.

[0071] The target data migration instruction is used to transfer the first target data in the target file from the high-performance storage end to the low-performance storage end for storage, and / or transfer the second target data in the target file from the low-performance storage end to the high-performance storage end for storage; the first target data includes the first data and the metadata corresponding to the first data; the second target data includes the second data and the metadata corresponding to the second data.

[0072] In this embodiment, the provided hierarchical data storage method is applied to the HSM (Hierarchical Storage Management) module in Lustre. Specifically, the HSM in Lustre is a technical module in the Lustre file system used to implement hierarchical data storage. It can provide a cost-effective storage platform that balances performance, capacity, and long-term retention (archiving) requirements, and optimizes the utilization of storage resources by migrating data between storage devices with different performance and cost.

[0073] "Classification" refers to dividing the data in the target file into storage tiers of different performance and cost levels based on the different characteristics of each data item, such as access frequency, business value, and timeliness. This means storing data on storage devices with different performance and cost. For example, frequently accessed data with high real-time requirements is stored on high-performance storage media, i.e., high-performance storage; such as parallel file systems and high-speed solid-state drives (SSDs). These types of storage have fast read and write speeds and can quickly respond to business requests. Conversely, less frequently accessed, historical, and archived data is stored on low-performance but inexpensive storage media, i.e., low-performance storage; such as object storage, NFS, and tape libraries.

[0074] In a tiered storage scenario, data within the same file may flow bidirectionally between different storage devices, and this bidirectional flow may also occur simultaneously. Taking object storage as an example, data may flow from the parallel distributed end to the object storage end, and simultaneously flow back from the object storage end to the parallel distributed end due to business needs. This bidirectional data flow is automatically performed based on factors such as data usage frequency and storage strategies.

[0075] Therefore, when the same file encounters bidirectional intelligent data flow simultaneously, data consistency issues may arise. For example, during the flow of data from local storage to object storage, if the local storage modifies the file while the object storage is also performing operations on the file (such as reading or writing based on other business logic), this may lead to inconsistencies in the file data status between the two storage devices. If left unaddressed, this will affect the accuracy and integrity of the data, thereby impacting the normal operation of the business. Therefore, in this embodiment, data synchronization will be performed through subsequent steps.

[0076] Specifically, when a data migration command is used to transfer the first target data in a target file from a high-performance storage device to a low-performance storage device, this operation is primarily based on a combination of storage cost and data access frequency. High-performance storage devices typically offer fast read and write speeds but have higher storage costs, making them suitable for storing frequently accessed data. Low-performance storage devices, while having relatively slower read and write speeds, have lower storage costs, making them suitable for storing data with lower access frequency. This migration optimizes the utilization efficiency of storage resources while ensuring data availability.

[0077] Conversely, when a target data migration instruction is used to transfer the second target data in a target file from a low-performance storage device to a high-performance storage device, it is generally because the access frequency of these data suddenly increases. In order to meet the business's demand for fast data reading and processing, it is necessary to migrate them to the high-performance storage device.

[0078] This application, recognizing the unique characteristics of different storage systems, utilizes data flow to store frequently accessed, performance-critical "hot" data in a higher-performance parallel file system. This ensures rapid access and processing of hot data, meeting business performance requirements. Simultaneously, infrequently accessed "cold" data is stored in lower-cost heterogeneous storage, fully leveraging the cost advantages of heterogeneous storage. This achieves an optimal balance between performance and cost, satisfying business data access performance requirements while effectively controlling storage costs.

[0079] It should be noted that the target data includes the primary data and its corresponding metadata. The primary data is the actual business data, while the metadata describes the attributes, characteristics, and source of the primary data, such as creation time, file format, and the user to which it belongs. Therefore, compared to data migration where only metadata is transmitted, the receiving end can obtain relevant descriptive information but lacks the actual data, making it unusable directly. This application transmits both the data itself and the metadata simultaneously. After migration, the receiving end can immediately process and analyze the data, greatly improving data availability and meeting real-time business requirements. For example, in financial transaction data migration scenarios, if only metadata is migrated, transaction records cannot be obtained in real time, affecting transaction analysis and decision-making; while migrating both data and metadata simultaneously allows financial institutions to promptly grasp transaction information and make accurate decisions.

[0080] It's important to note that in tiered storage scenarios, data within the same file may flow bidirectionally between different storage devices, and this bidirectional flow can occur simultaneously. Taking object storage as an example, data may flow from the parallel distributed storage to the object storage, and simultaneously flow back from the object storage to the parallel distributed storage due to business needs. This bidirectional data flow is automatically determined based on factors such as data usage frequency and storage strategies.

[0081] Therefore, when the same file encounters simultaneous bidirectional intelligent data flow, data consistency issues may arise. For example, during the flow of data from local storage to object storage, if the local storage modifies the file while the object storage is also performing operations on the same file (such as reads or writes based on other business logic), this could lead to inconsistencies in the file's data state between the two storage systems. If left unaddressed, this will affect the accuracy and integrity of the data, thereby impacting the normal operation of the business.

[0082] S102, in response to the target data migration instruction, migrate the first target data to a low-performance storage device for storage, and / or migrate the second target data to a high-performance storage device for storage.

[0083] Therefore, following step S102, the data migration operation needs to be performed immediately according to the target data migration instruction. Specifically, upon receiving the target data migration instruction, the system will first package and organize the data to be migrated. For example, when migrating the first target data from a high-performance storage device to a low-performance storage device, the metadata in the first target data will be checked for integrity and accuracy to ensure that no critical information is lost during the migration process. Simultaneously, the data to be migrated can be preprocessed, such as compressed or encrypted, to improve transmission efficiency and data security. If the migration is from a low-performance storage device to a high-performance storage device, the data format may also need to be converted to adapt to the requirements of the high-performance storage device.

[0084] It should be noted that after determining the direction of data migration based on the target data migration instructions, the system begins data transmission. During this process, data can be transmitted using appropriate transport protocols; for example, TCP can be used, which has error detection and retransmission mechanisms to effectively prevent data loss or corruption during transmission. For large-scale data migration, segmented transmission may be used, dividing the data into multiple data packets for sequential transmission to reduce network bandwidth consumption. Once the data is transmitted to the corresponding storage device, the system allocates an appropriate storage location. On high-performance storage devices, data is stored in a cache or a specific area of ​​a high-performance disk array based on its access frequency and importance; on low-performance storage devices, appropriate storage media, such as specific tapes in a tape library, is selected based on storage capacity and cost.

[0085] S103. After the migration of the first target data and / or the second target data is completed, generate the change information corresponding to the high-performance storage end and / or the low-performance storage end.

[0086] Specifically, once the data migration operation is complete—that is, once the first target data has been successfully migrated from the high-performance storage to the low-performance storage, or the second target data has been migrated from the low-performance storage to the high-performance storage, or both migrations have been completed—the system will automatically trigger a data change record generation program. This program will check and compare the data storage status of the high-performance and low-performance storage, extract the data change information, and record it according to certain formats and rules. This will immediately generate corresponding data change records for the high-performance and low-performance storage respectively.

[0087] Changelogs record all changes to files on both high-performance and low-performance storage, accurately describing the data's evolution. For high-performance storage, detailed information about the first or second target data being migrated may be recorded, such as filename, size, format, metadata (creation time, modification time, data owner, etc.), and the original storage location. For low-performance storage, changelogs record equally detailed information about newly migrated first or second target data, potentially including storage location allocation. For example, when migrating first target data from high-performance to low-performance storage, the high-performance storage loses some of its data, changing its storage status and requiring a changelog to be generated. Simultaneously, the low-performance storage adds this data, changing its storage content, necessitating a corresponding changelog. By continuously recording these changelogs, the system can monitor all changes to files on both storage ends.

[0088] Furthermore, these data change records facilitate subsequent data management and synchronization operations. Through these records, the system can clearly understand the differences in the state of the two storage devices after data migration, enabling targeted data synchronization to ensure data consistency between high-performance and low-performance storage devices. For example, at a later point in time, if data migration or data querying is required, the system can quickly and accurately locate the relevant data and understand its historical changes based on these data change records. In addition, data change records can also be used for data auditing and troubleshooting. When data inconsistencies or other problems occur, these records serve as important references, helping technical personnel quickly pinpoint the root cause of the problem.

[0089] S104. Synchronize data between high-performance storage and low-performance storage based on change information.

[0090] The change information includes the data change status and data change record status corresponding to the high-performance or low-performance storage end.

[0091] In some embodiments, tiered storage typically involves different types or performance levels of storage media, such as high-speed disk storage and large-capacity but relatively low-speed tape storage. During data migration or access, if the data at both ends is out of sync, some data may be missing at one end, resulting in data loss and impacting normal business operations and data integrity. Furthermore, different storage tiers may be accessed by different applications or business processes. If the data at both ends is inconsistent, different business processes will obtain different data, potentially leading to incorrect data analysis results, flawed business decisions, and other problems, severely impacting the company's operations and development.

[0092] Therefore, this application embodiment will synchronize data between the high-performance storage end and the low-performance storage end based on the change information; the change information includes the data change status and data change record status corresponding to the high-performance storage end or the low-performance storage end; thereby ensuring that the data at both ends is consistent.

[0093] Specifically, the change information includes the data change status and data change record status corresponding to the high-performance or low-performance storage end; the data change status indicates whether a data change has occurred on the current storage end; the data change record status indicates whether a data change record exists on the current storage end; in this embodiment, the data change status can be represented by 0 and 1, where 0 indicates no data change and 1 indicates a data change; the data change record status can also be represented by 0 and 1, where 0 indicates no data change record and 1 indicates a data change record.

[0094] Therefore, when performing data synchronization, it is necessary to synchronize data based on the data change status and data change record status corresponding to the high-performance storage end and the low-performance storage end, respectively.

[0095] Specifically, in the embodiment of this application, when synchronizing data between high-performance storage and low-performance storage based on change information, the following steps are also included:

[0096] It periodically reads change information corresponding to high-performance and low-performance storage from the database, and synchronizes the data between the high-performance and low-performance storage based on the change information.

[0097] Based on the above steps, during system operation, the system will periodically read the change information corresponding to the high-performance and low-performance storage ends to monitor data changes on both ends in real time, helping to promptly detect any file changes at both storage ends. Regardless of whether files on the high-performance or low-performance storage ends change, the system can use the change information to achieve comprehensive monitoring of the entire storage system.

[0098] Specifically, refer to Figure 2The diagram shown is the overall architecture of the hierarchical data storage method, which includes a high-performance storage terminal 21, a data flow control center 22 of the Lustre-HSM module, and a low-performance storage terminal 23.

[0099] High-performance storage device 21 is a storage device or storage area with high-speed read and write performance. It typically uses high-speed storage media, such as solid-state drives (SSDs), to store target data.

[0100] In contrast to high-performance storage, low-performance storage typically uses lower-cost storage media with relatively slow read / write speeds, such as tape libraries and hard disk drives. It is often used to store data that is not frequently accessed but needs to be preserved long-term, thereby reducing storage costs.

[0101] The data flow control center 22 of the Lustre-HSM module is used to manage and control the flow of data between the high-performance storage terminal 21 and the low-performance storage terminal 23.

[0102] When performing data migration using the above method, since the target data to be migrated includes the data itself and its corresponding metadata, the receiving end can directly process and analyze the data after the migration is complete, avoiding the problem of data being unusable due to only migrating metadata. Simultaneously, because corresponding change information is generated after data migration, different strategies are adopted during data synchronization based on the data change status and record status in the change information to synchronize data between high-performance and low-performance storage devices, thereby preventing data loss or overwriting and further improving system performance and stability.

[0103] As an extension and refinement of the above embodiments, this application also provides another hierarchical data storage method, referring to... Figure 3 As shown, the specific steps include the following S301 to S306:

[0104] S301. Monitor the access frequency of the target data and obtain the access frequency of the target data.

[0105] Specifically, Lustre's HSM module monitors data access in real time and triggers corresponding migration instructions based on preset rules, such as access count thresholds and access time intervals, to ensure that data is always in the most suitable storage tier, thereby improving the performance and efficiency of the entire storage system.

[0106] Access frequency refers to the number of times target data is read or written within a certain period. The system analyzes this access frequency data according to preset rules and strategies. For example, if it finds that some target data has a low access frequency on the high-performance storage side and has not been used for a long time, the system may generate instructions to migrate this data to the low-performance storage side to save high-performance storage resources; conversely, if the access frequency of some data on the low-performance storage side suddenly increases, the system may generate instructions to migrate it to the high-performance storage side to improve data access speed. In this way, dynamic and intelligent migration of data between different storage tiers is realized, optimizing the performance and resource utilization efficiency of the entire tiered storage system.

[0107] S302. When the target data is stored in a high-performance storage device and the access frequency is less than the first preset threshold; or when the target data is stored in a low-performance storage device and the access frequency is greater than the second preset threshold, a target data migration instruction is generated.

[0108] Specifically, since high-performance storage is typically more expensive but offers faster read and write speeds, making it suitable for storing frequently accessed hot data, when target data is stored on high-performance storage and its access frequency is less than a first preset threshold, it means that the target data is no longer hot data, and continuing to occupy high-performance storage resources would be wasteful. At this point, the system will generate an instruction to migrate the target data from high-performance storage to low-performance storage, freeing up high-performance storage resources for more needed hot data.

[0109] Low-performance storage typically offers large capacity and low cost, but its read / write speed is relatively slow, making it suitable for storing cold data. Therefore, if the target data is on high-performance storage and its access frequency exceeds a second preset threshold, and the access frequency of the target data originally stored on low-performance storage suddenly exceeds the preset threshold, it indicates that the data has changed from cold data to hot data. To meet the business's need for fast data access, the system will generate instructions to migrate the target data from low-performance storage to high-performance storage.

[0110] S303, Receive the target data migration instruction for the target file.

[0111] The target data migration instruction is used to transfer the first target data in the target file from the high-performance storage end to the low-performance storage end for storage, and / or transfer the second target data in the target file from the low-performance storage end to the high-performance storage end for storage; the first target data includes the first data and the metadata corresponding to the first data; the second target data includes the second data and the metadata corresponding to the second data.

[0112] The specific explanation of this step is as described in step S101 of the above embodiment, and will not be repeated here.

[0113] S304. In response to the target data migration instruction, migrate the first target data to a low-performance storage device for storage, and / or migrate the second target data to a high-performance storage device for storage.

[0114] The specific explanation of this step is as described in step S102 of the above embodiment, and will not be repeated here.

[0115] S305. After the migration of the first target data and / or the second target data is completed, generate the change information corresponding to the high-performance storage end and / or the low-performance storage end.

[0116] The specific explanation of this step is as described in step S103 of the above embodiment, and will not be repeated here.

[0117] S306. Periodically read the change information corresponding to the high-performance storage end and the low-performance storage end from the database, and synchronize the data between the high-performance storage end and the low-performance storage end based on the change information.

[0118] Specifically, the system will retrieve relevant data from the database at pre-set fixed time intervals, such as hourly, daily, or weekly; thus, it can continuously and regularly monitor data changes on the storage side, rather than retrieving information once or irregularly.

[0119] It's important to note that the core objective of data synchronization is to ensure data consistency between high-performance and low-performance storage systems. In tiered storage systems, data may differ across storage locations due to various operations (such as data migration, read / write operations by business systems, etc.). Data synchronization can eliminate these differences, ensuring the accuracy and integrity of data at both ends.

[0120] Therefore, this application periodically reads the data change records corresponding to the high-performance storage end and the low-performance storage end from the database to obtain all data changes that have occurred on the two storage ends, thereby facilitating the system to compare and analyze the data on the high-performance storage end and the low-performance storage end and to perform subsequent data synchronization.

[0121] Before synchronizing data based on the data change status and data change record status corresponding to the high-performance storage end and the low-performance storage end respectively, the method also includes the following steps one to three:

[0122] Step 1: Determine whether the data change status of both the high-performance storage end and the low-performance storage end is "no data change" and whether the data change record status is "no data change record".

[0123] Specifically, before synchronizing data, the system needs to obtain the status of the high-performance storage device and the low-performance storage device.

[0124] If both the high-performance and low-performance storage ends have no data change status (i.e., 0), it indicates that no data changes or data change records have been generated on either end within a certain period. Therefore, the data on both ends is synchronized and no data synchronization is required.

[0125] Step 2: If so, no data synchronization is required.

[0126] If the judgment result in step one is that the data change status of both the high-performance storage end and the low-performance storage end is 0, and the data change record status is also 0, this indicates that the actual content of the data on the two storage ends has not changed, and no data change has been recorded. In this case, it can be determined that the data on both ends is completely consistent, and no data synchronization operation is needed. This avoids unnecessary consumption of system resources, such as network bandwidth occupation and wasted computing resources, thereby improving the system's operating efficiency.

[0127] Step 3: If not, then perform data synchronization based on the data change status and data change record status corresponding to the high-performance storage end and the low-performance storage end, respectively.

[0128] If the judgment result in step one does not meet the condition that the data change status and data change record status of both storage ends are 0, it means that there may be inconsistencies between the data on the high-performance storage end and the low-performance storage end.

[0129] At this point, the system synchronizes data based on the specific data change status (0 or 1) and data change record status (0 or 1) of both the high-performance and low-performance storage terminals. By analyzing this status information, the system can determine the differences between the two sets of data, such as which data has been modified, added, or deleted. Then, based on these differences, the system transfers data from one storage terminal to the other, or modifies the data on both terminals accordingly, to ensure that the data on both terminals eventually remains consistent, thereby achieving the purpose of data synchronization.

[0130] Referring to the embodiments of this application, Figure 4 The diagram shows a flowchart illustrating the flow of data between high-performance and low-performance storage ends in the Lustre file system through hierarchical storage management, including:

[0131] The high-performance storage terminal 41 can be a parallel distributed storage terminal used for file creation, writing, reading, opening, appending, and deleting operations. This operation information is recorded in the high-performance storage terminal's change log, which records changes to the file system.

[0132] The low-performance storage end 42 can be an object storage system used for put, update, and delete operations. These operations are recorded in the "change log" of the low-performance storage end, which is the change log of the object storage.

[0133] Then, through the corresponding strategy mechanism, the change logs of the high-performance storage end and the low-performance storage end are written into database 44. Database 44 is used to temporarily store and manage this information. In practical applications, database 44 can also be a message queue, which can also be used to temporarily store and manage change logs.

[0134] It should be noted that in database 44, the query file needed for transfer can be used to determine which files need to be transferred between high-performance storage 41 and low-performance storage 42.

[0135] Event trigger 45 is used to trigger related operations. When the target data is stored in a high-performance storage device and the access frequency is less than the first preset threshold; or when the target data is stored in a low-performance storage device and the access frequency is greater than the second preset threshold, a target data migration instruction is generated. Event trigger 45 then triggers the hierarchical storage management system (HSM) to perform data migration operations. It can also trigger various operations including archiving, releasing, and restoring.

[0136] Ultimately, HSM will manage data across different storage tiers based on the received instructions. For example, it can migrate infrequently used data from the high-performance storage tier 41 to the low-cost storage tier 42 for archiving, or restore data from the low-performance storage tier 42 to the high-performance storage tier 41 when needed. This will optimize the utilization of storage resources and balance storage costs and performance requirements.

[0137] As an extension and refinement of the above embodiments, the specific implementation method of data synchronization based on the data change status and data change record status corresponding to the high-performance storage end and the low-performance storage end respectively includes the following:

[0138] Scenario 1: If the data change status of either the high-performance storage end or the low-performance storage end is "data changed", and the data change records of both the high-performance storage end and the low-performance storage end are "no data change", then a log replay instruction will be generated.

[0139] The log replay instruction is used to instruct the replay of the first log remnant records corresponding to the high-performance storage end and the low-performance storage end, respectively, so as to perform data synchronization based on the first log remnant records.

[0140] It's important to note that log remnants refer to situations where, during bidirectional data flow, one storage device's log records a file operation, but that operation hasn't yet been executed on the other storage device. For example, the high-performance storage device's log might record a file renaming operation, but due to network latency or other reasons, the low-performance storage device hasn't executed this renaming. In this case, because the log accurately records the data changes, it can be replayed normally. The system will rename the corresponding file on the low-performance storage device according to the renaming operation recorded in the high-performance storage device's log, thus achieving data synchronization between the two devices.

[0141] Specifically, when the data change status of either the high-performance or low-performance storage device shows as changed (i.e., data change status is 1), and both the high-performance and low-performance storage devices show no data change records (i.e., data change record status is 0), a log replay command is generated. The purpose of this command is to instruct the system to replay the first residual log record corresponding to each of the high-performance and low-performance storage devices. By replaying these residual log records, the system can perform data synchronization based on the information recorded in the logs. This is because even if no data change is recorded, if data changes have occurred, the logs may contain relevant data change information for synchronization.

[0142] Scenario 2: If the data change status of both the high-performance storage end and the low-performance storage end is "no data change", and the data change record status of either the high-performance storage end or the low-performance storage end is "no data change record", then the first data synchronization instruction is generated.

[0143] The first data synchronization instruction is used to instruct either the high-performance storage end or the low-performance storage end to perform data synchronization based on the first dirty data.

[0144] It's important to note that during the bidirectional data flow between high-performance and low-performance storage devices, some data changes may not be recorded in the logs. This unrecorded data is referred to as dirty data. For example, on the low-performance storage device, due to system failure or human error, the content of a file might be modified, but this modification might not be recorded in the logs. When this happens, since the system cannot automatically handle these changes based on the logs, a data synchronization command needs to be generated. The administrator will then manually synchronize the modified file data from the low-performance storage device to the high-performance storage device, or vice versa, to ensure data consistency between the two devices.

[0145] Specifically, when both the high-performance and low-performance storage devices show no data changes (i.e., a data change status of 0), and the data change record status of either the high-performance or low-performance storage device is also zero (i.e., no data change record status), a first data synchronization instruction is generated. This instruction instructs either the high-performance or low-performance storage device to synchronize data based on the first dirty data. Dirty data here refers to data that may contain inconsistencies or errors. Although the data change status shows no changes, the presence of dirty data necessitates synchronization to ensure data consistency.

[0146] Scenario 3: If the data change status of either the high-performance or low-performance storage end is "data change present" and the data change record status of either the high-performance or low-performance storage end is "data change present", then a second data synchronization instruction is generated; after the second data synchronization instruction is executed, a second log replay instruction is generated.

[0147] The second data synchronization instruction is used to instruct either the high-performance storage end or the low-performance storage end to perform data synchronization based on the second dirty data; the second log replay instruction is used to instruct the replay of the second log residual records corresponding to the high-performance storage end and the low-performance storage end respectively, so as to perform data synchronization based on the second log residual records.

[0148] It's important to note that if the high-performance storage contains corrupted data—for example, part of a file's data was accidentally deleted without being logged, while the low-performance storage retains log entries recording formatting operations on that file—directly replaying the low-performance storage's logs might overwrite the file's state on the high-performance storage, making the lost data unrecoverable. Therefore, a second data synchronization is necessary to first synchronize the actual data state (including the corrupted data) from the high-performance storage to the low-performance storage. Then, the remaining log entries on the low-performance storage are processed, and formatting adjustments are performed. This prevents the loss of corrupted data and ensures data integrity.

[0149] Specifically, when the data change status of either the high-performance or low-performance storage terminal is "data changed" (i.e., data change status is 1), and the data change record status of either the high-performance or low-performance storage terminal is "data changed record" (i.e., data change record status is 1), a second data synchronization instruction is first generated. This instruction instructs the high-performance or low-performance storage terminal to perform data synchronization based on the second dirty data. After executing the second data synchronization instruction, a second log replay instruction is generated. This instruction instructs the replay of the second log residual records corresponding to the high-performance and low-performance storage terminals, respectively, so as to perform further data synchronization based on these log residual records and ensure data accuracy and consistency.

[0150] Case 4: If both the high-performance storage and the low-performance storage have data change statuses, and both have no data change records, then a third data synchronization instruction is generated.

[0151] The third data synchronization instruction is used to instruct the high-performance storage end and the low-performance storage end to synchronize data according to the log version information of the fourth dirty data corresponding to their respective third dirty data.

[0152] It's important to note that when log entries remain on both the high-performance and low-performance storage ends, it indicates that both ends recorded different operations on the file. For example, the high-performance storage end might log modifications to the file content, while the low-performance storage end might log encryption operations. To ensure data consistency and synchronization to the latest state, the system needs to compare the version numbers and modification times of the data on both ends. The version number typically reflects the data's update iterations, while the modification time directly indicates the order of operations. By comprehensively comparing these two factors, the system can determine which storage end's operation is the most recent and synchronize the data on the other end according to the latest operation. For instance, if the high-performance storage end has a newer version and a later modification time, the low-performance storage end will be updated according to the high-performance storage end's operation, and vice versa.

[0153] Specifically, when both the high-performance and low-performance storage ends show a data change status (i.e., both are 1), and both have no data change records (i.e., both are 0), a third data synchronization instruction is generated. This instruction instructs the system to synchronize data based on the log version information of the third and fourth dirty data corresponding to the high-performance and low-performance storage ends, respectively. Because both ends have data changes but no change records, the log version information of the dirty data is used to determine and synchronize the data, ensuring correct data updates.

[0154] Case 5: If the data change status of both the high-performance storage end and the low-performance storage end is "no data change", and the data change record status of both the high-performance storage end and the low-performance storage end is "data change record", then a third log replay instruction is generated.

[0155] The third log replay instruction instructs the replay of log remnants based on the log version information of the third and fourth log remnants corresponding to the high-performance and low-performance storage ends, respectively, so as to synchronize data according to the log version information.

[0156] It's important to note that if both high-performance and low-performance storage devices contain dirty data, it means that changes to the data on both ends have not been correctly recorded and processed. For example, a file on the high-performance storage device might have been incorrectly modified in part, while the same file on the low-performance storage device might have had its formatting incorrectly set. In such complex situations, the system cannot automatically determine which storage device's data is more accurate, requiring manual intervention from the administrator. The administrator will carefully compare the version numbers and update times of the data on both ends to determine which version is the latest and correct. Then, based on the comparison results, a manual task will be issued to synchronize the latest and most accurate data to the other storage device, ensuring consistency and accuracy between the two ends.

[0157] Specifically, when both the high-performance and low-performance storage devices are in a "no data change" state (i.e., both are 0), and both have a "data change record" status (i.e., both are 1), a third log replay instruction is generated. This instruction instructs the system to replay the log remnants based on the log version information of the third and fourth log remnants for the high-performance and low-performance storage devices, respectively. By replaying the log remnants and synchronizing data based on the log version information, data consistency is ensured even though the data change status shows no change.

[0158] Case 6: If both the high-performance storage and low-performance storage have data change statuses, and the data change record status of either the high-performance or low-performance storage is "data change record", then a fourth data synchronization instruction is generated. After the fourth data synchronization instruction is executed, a fourth log synchronization instruction is generated.

[0159] The fourth data synchronization instruction is used to instruct the high-performance storage end and the low-performance storage end to synchronize data according to the log version information of the sixth dirty data of their respective fifth dirty data; the fourth log replay instruction is used to instruct the replay of the fifth log residual records corresponding to the high-performance storage end and the low-performance storage end, so as to synchronize data according to the fifth log residual records.

[0160] For example, both high-performance and low-performance storage devices have residual log records. The high-performance storage device records file permission modification operations, while the low-performance storage device records file content addition operations. Simultaneously, the low-performance storage device contains dirty data, with some added content containing errors. First, the system will suggest issuing a manual task to synchronize the actual data state containing dirty data from the low-performance storage device to the high-performance storage device. Then, based on the versions and modification times of the data on both devices, the system will automatically compare the permission modification and content addition operations to determine which is the latest, and synchronize the data on the other device according to the latest operation. This approach addresses the dirty data issue first, and then ensures that data updates are based on the latest operations.

[0161] Specifically, when both the high-performance and low-performance storage terminals are in a "data change has been made" state (i.e., both are in state 1), and either the high-performance or low-performance storage terminal has a "data change record has been made" state (i.e., state 1), a fourth data synchronization instruction is first generated. This instruction instructs the system to synchronize data based on the log version information of the fifth and sixth dirty data corresponding to the high-performance and low-performance storage terminals, respectively. After executing the fourth data synchronization instruction, a fourth log replay instruction is generated. This instruction instructs the system to replay the fifth residual log records corresponding to the high-performance and low-performance storage terminals, respectively, so as to perform data synchronization based on these residual log records and ensure data consistency and accuracy.

[0162] Case 7: If the data change status of either the high-performance storage end or the low-performance storage end is in the data change status, and the data change record status of both the high-performance storage end and the low-performance storage end is in the data change record, then the fifth data synchronization instruction is generated. After the fifth data synchronization instruction is executed, the fifth log synchronization instruction is generated.

[0163] Among them, the fifth data synchronization instruction is used to instruct the high-performance storage end or the low-performance storage end to perform data synchronization based on the seventh dirty data; the fifth log replay instruction instructs the replay of log residual records based on the log version information of the sixth log residual record and the seventh log residual record corresponding to the high-performance storage end and the low-performance storage end, respectively, so as to perform data synchronization based on the log version information.

[0164] Suppose both high-performance and low-performance storage devices contain corrupted data; some files on the high-performance storage device are missing data, while some files on the low-performance storage device are corrupted. Additionally, residual log records exist on the high-performance storage device, documenting file structure adjustments. First, the administrator needs to manually compare the data versions and update times on both devices to determine which storage device's data is more accurate. Assuming the high-performance storage device is determined to be more accurate, the administrator will issue a manual task to synchronize the data from the high-performance storage device to the low-performance storage device. Then, the logs from the high-performance storage device will be replayed normally, and file structure adjustments will be performed on the low-performance storage device. This process ensures data accuracy while completing data synchronization and updates.

[0165] Specifically, when the data change status of either the high-performance or low-performance storage terminal is "data changed" (i.e., data change status is 1), and both the high-performance and low-performance storage terminals have data change records (i.e., both have data change record status 1), a fifth data synchronization instruction is first generated. This instruction instructs either the high-performance or low-performance storage terminal to synchronize data based on the seventh dirty data. After executing the fifth data synchronization instruction, a fifth log replay instruction is generated. This instruction instructs the system to replay log remnants based on the log version information of the sixth and seventh log remnants corresponding to the high-performance and low-performance storage terminals, respectively. Data synchronization is performed based on log version information in this way to ensure data consistency.

[0166] Case 8: If both the high-performance storage and low-performance storage have data change statuses, and both have data change record statuses, then a sixth data synchronization instruction is generated. After the sixth data synchronization instruction is executed, a sixth log synchronization instruction is generated.

[0167] The sixth data synchronization instruction is used to instruct the high-performance storage end and the low-performance storage end to synchronize data according to the log version information of the ninth dirty data corresponding to the eighth dirty data, respectively; the sixth log replay instruction instructs the replay of log residual records according to the log version information of the eighth log residual record and the ninth log residual record corresponding to the high-performance storage end and the low-performance storage end, respectively, so as to synchronize data according to the log version information.

[0168] For example, both high-performance and low-performance storage systems may contain corrupted data. Files on the high-performance system may have distorted formats, while files on the low-performance system may have lost some content. Furthermore, the high-performance system's logs may record font modifications, while the low-performance system's logs may record paragraph adjustments. First, the administrator must manually compare the versions and update times of the data on both systems to determine which storage system's data is more accurate and complete. Assuming the high-performance system's data is superior, the administrator issues a manual task to synchronize the data from the high-performance system to the low-performance system. Then, based on the versions and change times of the data on both systems, the administrator automatically compares the font modification and paragraph adjustment operations to determine which is the latest, and synchronizes the other system's data according to the latest operation. This resolves the inconsistency caused by corrupted data first, and then further ensures data synchronization and updates to the latest state based on log records.

[0169] Specifically, when both the high-performance and low-performance storage ends are in a data change status (i.e., both are in a data change status of 1), and both have data change record statuses (i.e., both have data change record statuses of 1), a sixth data synchronization instruction is first generated. This instruction instructs the system to synchronize data based on the log version information of the eighth and ninth dirty data corresponding to the high-performance and low-performance storage ends, respectively. After executing the sixth data synchronization instruction, a sixth log replay instruction is generated. This instruction instructs the system to replay the log remnants based on the log version information of the eighth and ninth log remnants corresponding to the high-performance and low-performance storage ends, respectively. Finally, data synchronization is performed based on the log version information to ensure the consistency and accuracy of data at both ends.

[0170] Further referring to Table 1, the different situations of data change status and data change record status of the above-mentioned high-performance storage end and low-performance storage end can be further subdivided into 16 types as shown in Table 1 below;

[0171] Table 1

[0172]

[0173] As an extension and refinement of the above embodiments, given the deficiencies in the Coordinator and Copytool programs in the LustreHSM module—they do not provide high availability and high reliability—this means that during data management and migration, system failures and data inconsistencies may occur, affecting the stable operation of the entire storage system. Therefore, to improve the performance of the HSM module, this application embodiment further includes the following steps:

[0174] The Consul component is integrated into the native Lustre tiered storage management system to obtain the target tiered storage management system.

[0175] To address the shortcomings of the Lustre HSM module, the open-source component Consul was introduced without altering the native Lustre HSM mechanism. Consul is a tool for service discovery, configuration management, and distributed coordination. By integrating Consul into Lustre's tiered storage management system, an enhanced distributed task management system was built. Furthermore, in the Lustre HSM module of this application, the DataflowWorker module mimics the capabilities of the Coordinator in the original Lustre, responsible for task distribution and control. It can receive various data management tasks and allocate them appropriately to different processing units, ensuring efficient task execution.

[0176] The data flow system relies on its own reliability and high availability mechanisms to ensure data consistency before and after task execution. This means that during data migration, storage, and other operations, the system will take a series of measures, such as data verification and error recovery, to ensure that data is not lost or damaged, and remains accurate and complete at all times, thus optimizing the performance and resource utilization efficiency of the entire tiered storage system.

[0177] In conjunction with the above embodiments, the Consul component is integrated into the hierarchical storage management system within Lustre. Data flow worker nodes mimic the capabilities of the Coordiantor for task distribution and control. The data flow system relies on its own reliability and high availability mechanisms to ensure data consistency before and after task execution. (Refer to...) Figure 5 As shown, the specific flowchart includes:

[0178] Master node 51: It is used to undertake key coordination and decision-making tasks to ensure the orderly operation of the system. For example, it performs operations such as configuring and starting control of coordinator 56 to ensure that the coordinator can effectively coordinate the cooperation between various components.

[0179] Data flow management module 52: Connected to HTTP service module 53, it can communicate with external systems via HTTP protocol, receive external requests or provide data services, such as receiving file operation requests from clients; it is also connected to master node 51 and is responsible for electing the master node in the system; it is also connected to database 57 and is responsible for managing related data and storing this data in the database. This data may include system configuration information, task records, node status, etc., to support system management and monitoring.

[0180] Dataflow Worker 54: Used to manage and schedule data processing tasks in the system, such as the arrangement and execution monitoring of tasks like file creation, deletion, reading, and writing, as well as updating task file status (updating file status information during task execution to the database (DB) so that the system can monitor and manage task progress and file status); and creating or deleting files (creating or deleting files according to system needs). These operations may involve interaction with the metadata controller to ensure that the file's metadata information is updated accordingly.

[0181] The copy tool 55 is used for data copying operations. Specifically, under the control of the data flow management module 54, it performs operations such as reading, writing, and copying data. For example, when a file needs to be copied from one storage location to another, the copy tool 55 will execute the specific copying task.

[0182] The coordinator 56 is used to coordinate the work between various components in the system, such as the master node 51 and the replication tool 55, to ensure that they can work together and avoid conflicts or inconsistencies; and to obtain target metadata 59 when performing data operations.

[0183] Interact with target metadata 59: Interact with target metadata 59, and may be responsible for managing and maintaining the state of target metadata to ensure the accuracy and completeness of the metadata.

[0184] The metadata controller 58 is used for centralized management and control of metadata, such as file name, size, creation time, and access permissions. When the copy tool 55 performs data operations, the metadata controller 58 updates and maintains the metadata accordingly to ensure consistency between the metadata and the actual data.

[0185] Based on the same inventive concept, as an implementation of the above method, this application embodiment also provides a hierarchical data storage device. This embodiment corresponds to the aforementioned method embodiment. For ease of reading, this embodiment will not repeat the details of the aforementioned method embodiment one by one, but it should be clear that a hierarchical data storage device in this embodiment can correspondingly implement all the contents of the aforementioned method embodiment.

[0186] This application provides a hierarchical data storage device. Figure 6 This is a schematic diagram of the structure of the hierarchical data storage device, as shown below. Figure 6 As shown, the hierarchical data storage device 600 includes:

[0187] The receiving unit 601 is configured to receive a target data migration instruction for a target file; the target data migration instruction is configured to transfer first target data in the target file from a high-performance storage end to a low-performance storage end for storage, and / or transfer second target data in the target file from a low-performance storage end to a high-performance storage end for storage; the first target data includes first data and metadata corresponding to the first data; the second target data includes second data and metadata corresponding to the second data;

[0188] The migration unit 602 is configured to, in response to the target data migration instruction, migrate the first target data to the low-performance storage end for storage, and / or migrate the second target data to the high-performance storage end for storage;

[0189] The generation unit 603 is used to generate change information corresponding to the high-performance storage end and / or the low-performance storage end after the migration of the first target data and / or the second target data is completed;

[0190] The synchronization unit 604 is used to synchronize data between the high-performance storage terminal and the low-performance storage terminal based on the change information; the change information includes the data change status and data change record status corresponding to the high-performance storage terminal or the low-performance storage terminal.

[0191] As an optional implementation of this application, the generation unit 601 is specifically used to monitor the access frequency of the target data and obtain the access frequency of the target data; when the target data is stored in the high-performance storage terminal and the access frequency is less than a first preset threshold; or when the target data is stored in the low-performance storage terminal and the access frequency is greater than a second preset threshold, then the target data migration instruction is generated.

[0192] As an optional implementation of this application, the migration unit 602 is further configured to periodically read the change information corresponding to the high-performance storage terminal and the low-performance storage terminal from the database, and synchronize the data of the high-performance storage terminal and the low-performance storage terminal based on the change information.

[0193] As an optional implementation of this application, the migration unit 602 is further configured to perform data synchronization based on the data change status and data change record status corresponding to the high-performance storage end and the low-performance storage end, respectively.

[0194] As an optional implementation of this application, the synchronization unit 604 is specifically used to determine whether the data change status of the high-performance storage terminal and the low-performance storage terminal is both without data change, and whether the data change record status is both without data change record; if yes, then no data synchronization is required; if no, then data synchronization is performed based on the data change status and the data change record status corresponding to the high-performance storage terminal and the low-performance storage terminal respectively.

[0195] As an optional implementation of this application, the synchronization unit 604 is specifically used to generate a log replay instruction if the data change status of the high-performance storage terminal or the low-performance storage terminal is "data changed" and the data change records of the high-performance storage terminal and the low-performance storage terminal are both "no data change" records; the log replay instruction is used to instruct the replay of the first log residual records corresponding to the high-performance storage terminal and the low-performance storage terminal respectively, so as to perform data synchronization based on the first log residual records;

[0196] If the data change status of both the high-performance storage terminal and the low-performance storage terminal is "no data change", and the data change record status of either the high-performance storage terminal or the low-performance storage terminal is "no data change record", then a first data synchronization instruction is generated; the first data synchronization instruction is used to instruct the high-performance storage terminal or the low-performance storage terminal to perform data synchronization based on the first dirty data.

[0197] If the data change status of the high-performance storage terminal or the low-performance storage terminal is "data changed", and the data change record status of the high-performance storage terminal or the low-performance storage terminal is "data changed record", then a second data synchronization instruction is generated; after the second data synchronization instruction is executed, a second log replay instruction is generated; the second data synchronization instruction is used to instruct the high-performance storage terminal or the low-performance storage terminal to perform data synchronization based on the second dirty data; the second log replay instruction is used to instruct the replay of the second log residual records corresponding to the high-performance storage terminal and the low-performance storage terminal respectively, so as to perform data synchronization based on the second log residual records;

[0198] If both the high-performance storage terminal and the low-performance storage terminal have data change statuses, and both the high-performance storage terminal and the low-performance storage terminal have no data change records, then a third data synchronization instruction is generated. The third data synchronization instruction is used to instruct the high-performance storage terminal and the low-performance storage terminal to perform data synchronization based on the log version information of the fourth dirty data corresponding to their respective third dirty data.

[0199] If the data change status of both the high-performance storage terminal and the low-performance storage terminal is "no data change", and the data change record status of both the high-performance storage terminal and the low-performance storage terminal is "data change record", then a third log replay instruction is generated. The third log replay instruction instructs the replay of log residual records based on the log version information of the third log residual record and the fourth log residual record corresponding to the high-performance storage terminal and the low-performance storage terminal, respectively, so as to perform data synchronization based on the log version information.

[0200] If both the high-performance storage terminal and the low-performance storage terminal have data change statuses, and the data change record status of either the high-performance storage terminal or the low-performance storage terminal is "data change record", then a fourth data synchronization instruction is generated. After the fourth data synchronization instruction is executed, a fourth log synchronization instruction is generated. The fourth data synchronization instruction is used to instruct the high-performance storage terminal and the low-performance storage terminal to perform data synchronization based on the log version information of the sixth dirty data of their respective fifth dirty data. The fourth log replay instruction is used to instruct the replay of the fifth log residual records corresponding to the high-performance storage terminal and the low-performance storage terminal, so as to perform data synchronization based on the fifth log residual records.

[0201] If the data change status of either the high-performance storage terminal or the low-performance storage terminal has a data change status, and the data change record status of both the high-performance storage terminal and the low-performance storage terminal has a data change record, then a fifth data synchronization instruction is generated. After the fifth data synchronization instruction is executed, a fifth log synchronization instruction is generated. The fifth data synchronization instruction is used to instruct the high-performance storage terminal or the low-performance storage terminal to perform data synchronization based on the seventh dirty data. The fifth log replay instruction instructs the replay of log residual records based on the log version information of the sixth log residual record and the seventh log residual record corresponding to the high-performance storage terminal and the low-performance storage terminal, respectively, so as to perform data synchronization based on the log version information.

[0202] If both the high-performance storage terminal and the low-performance storage terminal have data change statuses, and both have data change record statuses, a sixth data synchronization instruction is generated. After the sixth data synchronization instruction is executed, a sixth log synchronization instruction is generated. The sixth data synchronization instruction is used to instruct the high-performance storage terminal and the low-performance storage terminal to synchronize data according to the log version information of their respective eighth dirty data and ninth dirty data. The sixth log replay instruction instructs the replay of log residual records according to the log version information of the eighth log residual records and the ninth log residual records corresponding to the high-performance storage terminal and the low-performance storage terminal, respectively, so as to synchronize data according to the log version information.

[0203] As an optional implementation of this application, the generation unit 601 is further configured to mount the Consul component in the native Lustre hierarchical storage management system to obtain the target hierarchical storage management system.

[0204] Based on the same inventive concept, this disclosure also provides an electronic device. Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this disclosure, such as... Figure 7 As shown, the electronic device provided in this embodiment includes a memory 701 and a processor 702. The memory 701 is used to store a computer program; the processor 702 is used to execute the audio data processing method provided in the above embodiment when executing the computer program.

[0205] Based on the same inventive concept, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the computing device to implement the hierarchical data storage method provided in the above embodiments.

[0206] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0207] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0208] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0209] Computer-readable media include both permanent and non-permanent, removable and non-removable storage media. Storage media can store information using any method or technology; the information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0210] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A hierarchical data storage method, characterized in that, The target tiered storage management system used in Lustre includes: The system receives a target data migration instruction for a target file; the target data migration instruction is used to transfer first target data in the target file from a high-performance storage end to a low-performance storage end for storage, and / or transfer second target data in the target file from a low-performance storage end to a high-performance storage end for storage; the first target data includes first data and metadata corresponding to the first data; the second target data includes second data and metadata corresponding to the second data; In response to the target data migration instruction, the first target data is migrated to the low-performance storage terminal for storage, and / or the second target data is migrated to the high-performance storage terminal for storage; After the migration of the first target data and / or the second target data is completed, change information corresponding to the high-performance storage terminal and / or the low-performance storage terminal is generated; the change information includes the data change status and data change record status corresponding to the high-performance storage terminal or the low-performance storage terminal. Data synchronization is performed based on the data change status corresponding to the high-performance storage end and the low-performance storage end, respectively, and the data change record status. The data synchronization based on the data change status corresponding to the high-performance storage terminal and the low-performance storage terminal, and the data change record status, includes: Determine whether the data change status of both the high-performance storage terminal and the low-performance storage terminal is "no data change" and whether the data change record status is "no data change record". If so, then data synchronization is not required; If not, data synchronization is performed based on the data change status and data change record status corresponding to the high-performance storage end and the low-performance storage end, respectively. The data synchronization based on the data change status corresponding to the high-performance storage terminal and the low-performance storage terminal, and the data change record status, includes: If both the high-performance storage terminal and the low-performance storage terminal have data change status, and the data change record status of either the high-performance storage terminal or the low-performance storage terminal has data change record, then a fourth data synchronization instruction is generated. After the fourth data synchronization instruction is executed, a fourth log replay instruction is generated. The fourth data synchronization instruction is used to instruct the high-performance storage terminal and the low-performance storage terminal to perform data synchronization based on the log version information of their respective fifth dirty data and sixth dirty data. The fourth log replay instruction is used to instruct the replay of the fifth log residual records corresponding to the high-performance storage terminal and the low-performance storage terminal, so as to perform data synchronization based on the fifth log residual records.

2. The method according to claim 1, characterized in that, Before receiving the target data migration instruction for the target file, the method further includes: Monitor the access frequency of the target data and obtain the access frequency of the target data; When the target data is stored in the high-performance storage device and the access frequency is less than a first preset threshold; or when the target data is stored in the low-performance storage device and the access frequency is greater than a second preset threshold, the target data migration instruction is generated.

3. The method according to claim 1, characterized in that, After the migration of the first target data and / or the second target data is completed, change information corresponding to the high-performance storage terminal and / or the low-performance storage terminal is generated, including: The system periodically reads the change information corresponding to the high-performance storage terminal and the low-performance storage terminal from the database, and synchronizes the data between the high-performance storage terminal and the low-performance storage terminal based on the change information.

4. The method according to claim 1, characterized in that, The data synchronization based on the data change status corresponding to the high-performance storage terminal and the low-performance storage terminal, and the data change record status, includes: If the data change status of the high-performance storage terminal or the low-performance storage terminal is "data changed", and the data change records of the high-performance storage terminal and the low-performance storage terminal are both "no data change records", then a first log replay instruction is generated; the first log replay instruction is used to instruct the replay of the first log residual records corresponding to the high-performance storage terminal and the low-performance storage terminal respectively, so as to perform data synchronization based on the first log residual records; If the data change status of both the high-performance storage terminal and the low-performance storage terminal is "no data change", and the data change record status of either the high-performance storage terminal or the low-performance storage terminal is "no data change record", then a first data synchronization instruction is generated; the first data synchronization instruction is used to instruct the high-performance storage terminal or the low-performance storage terminal to perform data synchronization based on the first dirty data. If the data change status of the high-performance storage terminal or the low-performance storage terminal is "data changed", and the data change record status of the high-performance storage terminal or the low-performance storage terminal is "data changed record", then a second data synchronization instruction is generated; after the second data synchronization instruction is executed, a second log replay instruction is generated; the second data synchronization instruction is used to instruct the high-performance storage terminal or the low-performance storage terminal to perform data synchronization based on the second dirty data; the second log replay instruction is used to instruct the replay of the second log residual records corresponding to the high-performance storage terminal and the low-performance storage terminal respectively, so as to perform data synchronization based on the second log residual records; If both the high-performance storage terminal and the low-performance storage terminal have data change status, and both have no data change record status, then a third data synchronization instruction is generated. The third data synchronization instruction is used to instruct the high-performance storage terminal and the low-performance storage terminal to perform data synchronization based on the log version information of their respective third dirty data and fourth dirty data. If the data change status of both the high-performance storage terminal and the low-performance storage terminal is "no data change", and the data change record status of both the high-performance storage terminal and the low-performance storage terminal is "data change record", then a third log replay instruction is generated. The third log replay instruction is used to instruct the replay of log residual records based on the log version information of the third log residual record and the fourth log residual record corresponding to the high-performance storage terminal and the low-performance storage terminal, respectively, so as to perform data synchronization based on the log version information. If the data change status of either the high-performance storage terminal or the low-performance storage terminal is "data changed", and the data change record status of both the high-performance storage terminal and the low-performance storage terminal is "data changed", then a fifth data synchronization instruction is generated. After the fifth data synchronization instruction is executed, a fifth log replay instruction is generated. The fifth data synchronization instruction is used to instruct the high-performance storage terminal or the low-performance storage terminal to perform data synchronization based on the log version information of the seventh dirty data. The fifth log replay instruction is used to instruct the replay of log residual records based on the log version information of the sixth and seventh log residual records corresponding to the high-performance storage terminal and the low-performance storage terminal, respectively, so as to perform data synchronization based on the log version information. If both the high-performance storage terminal and the low-performance storage terminal have data change status, and both have data change record status, a sixth data synchronization instruction is generated. After the sixth data synchronization instruction is executed, a sixth log replay instruction is generated. The sixth data synchronization instruction is used to instruct the high-performance storage terminal and the low-performance storage terminal to synchronize data according to the log version information of their respective eighth and ninth dirty data. The sixth log replay instruction is used to instruct the replay of log residual records according to the log version information of the eighth and ninth log residual records corresponding to the high-performance storage terminal and the low-performance storage terminal, respectively, so as to synchronize data according to the log version information.

5. The method according to claim 1, characterized in that, Before receiving the target data migration instruction for the target file, the method further includes: The Consul component is integrated into the native Lustre tiered storage management system to obtain the target tiered storage management system.

6. A hierarchical data storage device, characterized in that, include: The receiving unit is used to receive target data migration instructions for the target file; The target data migration instruction is used to transfer the first target data in the target file from a high-performance storage terminal to a low-performance storage terminal for storage, and / or to transfer the second target data in the target file from a low-performance storage terminal to a high-performance storage terminal for storage; the first target data includes first data and metadata corresponding to the first data; the second target data includes second data and metadata corresponding to the second data; A migration unit is configured to, in response to the target data migration instruction, migrate the first target data to the low-performance storage terminal for storage, and / or migrate the second target data to the high-performance storage terminal for storage; The generation unit is used to generate change information corresponding to the high-performance storage end and / or the low-performance storage end after the migration of the first target data and / or the second target data is completed. The change information includes the data change status and data change record status corresponding to the high-performance storage terminal or the low-performance storage terminal; A synchronization unit is used to perform data synchronization based on the data change status corresponding to the high-performance storage terminal and the low-performance storage terminal, and the data change record status, respectively. The synchronization unit is specifically used to determine whether the data change status of both the high-performance storage terminal and the low-performance storage terminal is "no data change" and whether the data change record status is "no data change record". If so, no data synchronization is required; if not, data synchronization is performed based on the data change status and data change record status corresponding to the high-performance storage terminal and the low-performance storage terminal, respectively. The synchronization unit is specifically used to generate a fourth data synchronization instruction if the data change status of both the high-performance storage terminal and the low-performance storage terminal is a data change status, and the data change record status of either the high-performance storage terminal or the low-performance storage terminal is a data change record. After the fourth data synchronization instruction is executed, a fourth log playback instruction is generated. The fourth data synchronization instruction is used to instruct the high-performance storage end and the low-performance storage end to perform data synchronization based on the log version information of their respective fifth dirty data and sixth dirty data; the fourth log replay instruction is used to instruct the replay of the fifth log residual records corresponding to the high-performance storage end and the low-performance storage end, so as to perform data synchronization based on the fifth log residual records.

7. An electronic device, characterized in that, include: A memory and a processor, the memory being used to store a computer program; the processor being used to cause the electronic device to implement the hierarchical data storage method according to any one of claims 1-5 when executing the computer program.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a computing device, causes the computing device to implement the hierarchical data storage method according to any one of claims 1-5.

Images