A cold data migration method and device, computer equipment and storage medium

Abstract

The application relates to the technical field of servers and discloses a cold data migration method and device, which comprises the following steps: obtaining a migration task and a plurality of directories of a proximal cluster; according to at least one cold data associated with a plurality of address pairs in the migration task, searching the plurality of directories to find whether a first directory exists and whether the first directory is mounted with a migration node; if the first directory is found, storing the address pairs associated with the at least one cold data in at least one first directory corresponding to the at least one cold data, so that each migration node hung on the first directory migrates the cold data corresponding to the address pairs from a proximal cluster address to a destination address of a distal cluster when detecting that one address pair is stored in the first directory. The application can effectively reduce errors that may occur in the migration process and ensure high accuracy of data migration.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of servers, and in particular to a cold data migration method and device, computer equipment and a storage medium. BACKGROUND

[0002] At present, in the design of modern distributed storage systems, with the continuous expansion of data volume and business scale, how to effectively reduce costs while ensuring performance has become an important challenge. Especially when dealing with massive unstructured data, the storage scheme of the system needs to have flexibility and scalability in order to cope with the growing storage demand. Massive unstructured data generally includes documents, pictures, videos, logs and other files. These data often have no fixed structure, and the update frequency and access mode are different. Low-frequency access or long-term non-update data is cold data. With the continuous increase of data volume, reasonable management of cold data has become one of the key strategies to optimize storage resources and improve storage efficiency. SUMMARY

[0003] Therefore, the present application provides a cold data migration method, device, computer equipment and storage medium to solve the problem of how to migrate cold data.

[0004] In a first aspect, the present application provides a cold data migration method, which is applied to a near-end cluster, and the method comprises:

[0005] Obtaining a migration task and a plurality of directories of a near-end cluster, the migration task comprising a plurality of address pairs, each address pair being associated with a cold data, and each address pair comprising a near-end cluster address and a remote cluster address, the near-end cluster address storing the cold data, and the remote cluster address being a destination address of the cold data to be migrated from the near-end cluster address;

[0006] According to at least one cold data associated with the plurality of address pairs in the migration task, searching whether a first directory exists in the plurality of directories and whether the first directory is mounted with a migration node;

[0007] If the first directory is found, storing the address pair associated with the at least one cold data in at least one first directory corresponding to the at least one cold data, so that each migration node hung on the first directory migrates the cold data corresponding to the address pair from the near-end cluster address to the destination address of the remote cluster when detecting that the address pair is stored in the first directory.

[0008] The application provides a cold data migration method. According to the association between the cold data and the address pair, the cold data is stored in a corresponding first directory, and the migration task related to the cold data can be quickly found. By accurately searching the first directory containing the migration node, it can be ensured that each cold data can be correctly migrated. Each migration task corresponds to specific cold data and an address pair, so that the task management is more clear and orderly. When at least one migration node obtains the address pair in the mounted first directory, the cold data migration task is automatically started. The manual intervention is reduced, and the reliability and accuracy of the migration process are improved. By ensuring the smooth migration of data from the near-end cluster to the far-end cluster, the allocation of storage resources is optimized, which helps to reduce the storage pressure of the cold data in the system. By using the directory structure and the address pair storage mechanism in the system, errors that may occur in the migration process can be effectively reduced, and the high accuracy of data migration is ensured.

[0009] In an optional embodiment, the address pair associated with the at least one cold data is stored in the at least one first directory corresponding to the at least one cold data.

[0010] A plurality of second directories are established, at least one corresponding relationship between the plurality of second directories and at least one cold data is established, and a corresponding relationship between the plurality of second directories and at least one first directory is established, wherein each second directory corresponds to a first directory and a cold data.

[0011] According to the plurality of second directories and the at least one corresponding relationship, the address pair corresponding to each cold data is stored in the second directory corresponding to each cold data, and each second directory stores a corresponding address pair.

[0012] The number of cold data in the migration task and the number of second directories in which the address pair is stored are obtained.

[0013] If the number of cold data and the number of second directories are the same, according to the plurality of second directories and the corresponding relationship between the plurality of second directories and at least one first directory, each address pair in each second directory is stored in each first directory.

[0014] The method provided in the embodiment avoids the incomplete address pair caused by the direct reading of the address pair by the migration node in the process of storing the address pair in the first directory, improves the organization and management efficiency of the cold data migration, optimizes the utilization of storage resources, reduces the storage pressure, realizes the automatic migration process, reduces the manual intervention, improves the scalability and flexibility of the system, and ensures the accuracy and efficiency of the data migration process.

[0015] In an optional implementation, the method further comprises:

[0016] generating a plurality of first indexes, each of the first indexes being associated with an address pair;

[0017] storing each of the first indexes and each of the address pairs in each of the first directories.

[0018] The method provided by the embodiment can quickly locate and access cold data by associating the storage addresses of cold data with first indexes. The storage locations of each cold data are accurately identified and indexed, avoiding the long search time that may occur in the traditional method. The use of directories to store index and address pairs makes the management of cold data more systematic and structured. Cold data usually has a low access frequency, but this method can ensure that cold data can be quickly obtained when needed.

[0019] In an optional implementation, the method further comprises:

[0020] obtaining the number of cold data in the migration task and the number of first directories storing the address pairs;

[0021] comparing the number of cold data with the number of first directories storing the address pairs, and if the number of cold data is equal to the number of first directories, modifying each of the first indexes to a corresponding second index in the first directories, each of the first indexes corresponding to one of the second indexes, each of the second indexes corresponding to one of the migration nodes, and each of the second indexes being used to instruct each of the corresponding migration nodes to obtain the associated address pair.

[0022] The method provided by the embodiment optimizes the processing of cold data migration by introducing the concepts of second indexes and migration nodes. When the number of cold data matches the number of first directories, the system converts the first indexes to second indexes, thereby assigning the migration task of cold data to migration nodes, improving the efficiency, flexibility and scalability of the migration process. At the same time, the design of the migration nodes enhances the integrity and consistency of the data, ensuring efficient access and reliability during the migration of cold data.

[0023] In an optional implementation, the method further comprises:

[0024] migrating the cold data from the near-end cluster address to the destination address of the far-end cluster in batches according to the splitting of the cold data into a plurality of data objects;

[0025] In the process of migrating the cold data in batches from the proximal cluster address to the destination address of the remote cluster according to the splitting into multiple data objects, further comprising:

[0026] If the update request or access time access is received, the current migration operation of the cold data is stopped, and it is determined that the data objects that have completed migration are on the destination address of the remote cluster;

[0027] The data objects that have completed migration are sent to the proximal cluster address, and after combining the data objects that have completed migration with the data objects that have not been migrated, the update or access operation is performed on the complete cold data.

[0028] The method provided by the embodiment introduces data splitting, batch migration, interruption mechanism of update request, and recovery operation after migration completion, so that the system can more smoothly perform the migration task of the cold data, ensures the high availability and consistency of the cold data in the migration process, reduces the risk of data loss, and improves the fault tolerance capability of the system when facing high load and faults. These technical effects make the data migration more efficient, reliable, and better support large-scale and complex cold data migration tasks.

[0029] In an optional implementation, each of the cold data includes multiple objects.

[0030] When migrating the corresponding cold data of the address pair from the proximal cluster address to the destination address of the remote cluster, further comprising:

[0031] Obtaining the total number of objects of each of the cold data, and the number of migrated objects in each of the cold data migrated to the remote cluster;

[0032] Comparing the total number of objects of each of the cold data with the number of migrated objects to obtain a second comparison result of each of the cold data;

[0033] According to the second comparison result of each of the cold data, determining a second target cold data from the multiple cold data, the total number of objects of the second target cold data is the same as the number of migrated objects;

[0034] Generating a migration completion identifier, the migration completion identifier indicating that the migration task corresponding to the second target cold data is completed.

[0035] The method provided in this embodiment can more accurately track the migration progress of each cold data during the migration process by obtaining the total number of objects and the number of migration objects for each cold data and comparing them. By clearly comparing the number of migration objects and the total number of objects, omissions or duplications during the migration process can be effectively avoided. By analyzing the second comparison result of each cold data, the priority and resource allocation of the migration task can be intelligently determined. By generating a migration completion identifier to indicate the completion of the migration task of the second target cold data, the system can efficiently mark the completion status of each migration task. By comparing the total number of objects and the number of migration objects, the system can accurately determine the progress of each cold data migration task and avoid unnecessary waste of resources.

[0036] In an optional implementation, after generating the migration completion mark, the process further includes:

[0037] Obtain metadata corresponding to each of the cold data, the metadata including: a storage address of the cold data;

[0038] According to each of the remote cluster addresses, metadata corresponding to each of the cold data is updated, where the metadata is used to access or modify the cold data corresponding to the read / write operation when processing the read / write operation.

[0039] The method provided in this embodiment obtains metadata corresponding to each cold data item and updates it based on the remote cluster address. This ensures that read and write operations after data migration accurately access the data in the remote cluster. This metadata update reflects the cold data migration status in real time, ensuring that all system components are aware of the cold data migration progress. By promptly updating metadata, the system can avoid invalid accesses caused by incorrect storage or cluster addresses. Precise metadata management improves the efficiency and transparency of cold data lifecycle management.

[0040] In a second aspect, the present invention provides a cold data migration device, which is applied to a near-end cluster and includes:

[0041] An acquisition module is configured to acquire a migration task and multiple directories of a near-end cluster, wherein the migration task includes multiple address pairs, each of which is associated with a piece of cold data, and each of which includes a near-end cluster address and a far-end cluster address, wherein the near-end cluster address stores the cold data, and the far-end cluster address is the destination address to which the cold data is to be migrated from the near-end cluster address;

[0042] a search module configured to search, based on at least one cold data associated with the multiple address pairs in the migration task, whether a first directory exists in the multiple directories and a migration node is mounted on the first directory;

[0043] The storage module is configured to, if the first directory exists, store the at least one address pair associated with the cold data into the at least one first directory corresponding to the at least one cold data, so that each migration node mounted on the first directory migrates the cold data corresponding to the address pair from the near-end cluster address to the destination address of the far-end cluster when detecting that the address pair is stored in the first directory.

[0044] The migration device for cold data provided by the application can quickly find the migration task related to the cold data according to the association between the cold data and the address pair and store the association into the corresponding first directory. The first directory containing the migration node can be accurately found, so that each cold data can be correctly supported by the migration. Each migration task corresponds to specific cold data and address pair, so that the task management is clearer and more orderly. When at least one migration node obtains the address pair in the mounted first directory, the cold data migration task is automatically started. The manual intervention is reduced, and the reliability and accuracy of the migration process are improved. The smooth migration of the data from the near-end cluster to the far-end cluster is ensured, the allocation of the storage resources is optimized, and the storage pressure of the cold data in the system is reduced. The directory structure and the address pair storage mechanism are used in the system, so that the errors in the migration process can be effectively reduced, and the high accuracy of the data migration is ensured.

[0045] In a third aspect, the application provides a computer device, comprising a memory and a processor, the memory and the processor are in communication connection with each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the migration method for cold data according to the first aspect or any of the corresponding embodiments thereof.

[0046] In a fourth aspect, the application provides a computer readable storage medium, which stores computer instructions, and the computer instructions are used to make the computer execute the migration method for cold data according to the first aspect or any of the corresponding embodiments thereof.

[0047] In a fifth aspect, the application provides a computer program product, which comprises computer instructions, and the computer instructions are used to make the computer execute the migration method for cold data according to the first aspect or any of the corresponding embodiments thereof. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the specific embodiments of the application or the technical solutions in the prior art, the drawings needed in the specific embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0049] Figure 1 This is a schematic diagram of a scenario architecture for cold data migration according to an embodiment of the present invention;

[0050] Figure 2 This is one of the flow charts of the cold data migration method according to an embodiment of the present invention;

[0051] Figure 3 This is a second flowchart of the cold data migration method according to an embodiment of the present invention;

[0052] Figure 4 is a schematic diagram of a cold data migration device according to an embodiment of the present invention;

[0053] Figure 5 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0055] According to an embodiment of the present invention, an embodiment of a cold data migration method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0056] Figure 1 Schematic diagram of a cold data migration scenario according to an embodiment of the present invention. Figure 1 As shown, the architecture includes: a near-end cluster, a far-end cluster, and a client.

[0057] The proximal cluster includes a migration node, a file system, and a user node. The proximal cluster can store and process hot data with high frequency access and high real-time requirement, is close to the user in geographical location, has low network delay, and has a high-performance storage system. The proximal cluster is connected with the client and the distal cluster respectively. The file system includes a plurality of directories, for example, a first directory 1, a first directory 2, and a first directory 3. The migration node is mounted under the plurality of first directories and is used to read an address pair stored in the first directory, for example, the migration node 1 is mounted under the first directory 1, the migration node 2 is mounted under the first directory 2, and the migration node 3 is mounted under the first directory 3. The user node is used to connect with the client and receive a migration task sent by the user through the client. The migration node and the user node can be the same node, and the file system can also be deployed on the node. The distal cluster is used to store cold data of the proximal cluster.

[0058] In the embodiment, a cold data migration method is provided, which can be used for a proximal cluster, Figure 2 is one of flowcharts of the cold data migration method according to an embodiment of the present application, as shown in Figure 2 The method comprises the following steps.

[0059] In step S201, a migration task and a plurality of directories of a proximal cluster are obtained.

[0060] The migration task includes a plurality of address pairs, each address pair is associated with a cold data, and each address pair includes a proximal cluster address and a distal cluster address. The cold data is stored on the proximal cluster address, and the distal cluster address is a destination address of the cold data to be migrated from the proximal cluster address.

[0061] Each address pair includes a corresponding address for storing the cold data in the proximal cluster and a storage address to which the cold data is to be migrated in the distal cluster. The cold data in the proximal cluster or the distal cluster can be stored in one address or segmented and stored in a plurality of addresses, that is, one cold data can correspond to a plurality of proximal cluster addresses or distal cluster addresses, and the plurality of addresses can be associated through a unique identity of the cold data.

[0062] One obtaining method is that the user node connected with the client receives the migration task sent by the client, and obtains the plurality of directories from the file system of the proximal cluster.

[0063] In step S202, whether there is a first directory in which a migration node is mounted in the plurality of directories is determined according to at least one cold data associated with the plurality of address pairs in the migration task.

[0064] One cold data corresponds to one first directory. The plurality of directories in the file system can be directories mounted with the migration node, or directories not mounted with the migration node, can be directories corresponding to the cold data, or directories not corresponding to the cold data.

[0065] One search method is to read the identification information of each directory. The identification information is preset information indicating whether the directory is mounted with the migration node and the corresponding cold data, or the corresponding first directory of each cold data can be searched from the plurality of directories according to the correspondence between the cold data and the first directory.

[0066] Further, if the first directory of the cold data is not found, a directory not corresponding to any cold data can be selected from the plurality of directories as the first directory corresponding to the cold data. If the first directory corresponding to the cold data is not mounted with the migration node, a migration node can be selected from the migration nodes not mounted under the directory and mounted under the first directory.

[0067] In step S203, if the first directory is found, the address pair of at least one cold data is stored in at least one first directory corresponding to at least one cold data, so that each migration node mounted under the first directory migrates the cold data corresponding to the address pair from the near-end cluster address to the destination address of the far-end cluster when detecting that the address pair is stored in the first directory.

[0068] One first directory stores the address pair of one corresponding cold data. The correspondence between the cold data and the first directory can be preset, or can be carried in the migration task, and the migration directory of each cold data specified by the user.

[0069] One storage method is to analyze the correspondence between the cold data and the first directory, determine the first directory corresponding to each cold data, and store the address pair corresponding to each cold data in the first directory corresponding to the cold data.

[0070] The address pair includes the near-end cluster address currently storing the corresponding cold data and the far-end cluster address to store the cold data.

[0071] One migration method is that the migration node reads the address pair in the first directory mounted thereby, analyzes the near-end cluster address and the far-end cluster address in the address pair, extracts the corresponding cold data from the near-end cluster address, and then stores the cold data in the corresponding far-end cluster address in the far-end cluster.

[0072] The above steps can be performed by one or more nodes of the near-end cluster, which is not limited by the present application.

[0073] The application provides a cold data migration method, according to the correspondence between the cold data and the address pair, and the cold data is stored in the corresponding first directory, so that the migration task related to the cold data can be quickly found. By accurately searching the first directory containing the migration node, it can be ensured that each cold data can be correctly migrated. Each migration task corresponds to a specific cold data and address pair, so that the task management is more clear and orderly. When at least one migration node obtains the address pair in the mounted first directory, the cold data migration task is automatically started. The manual intervention is reduced, and the reliability and accuracy of the migration process are improved. By ensuring the smooth migration of data from the near-end cluster to the far-end cluster, the allocation of storage resources is optimized, which helps to reduce the storage pressure of cold data in the system. By using the directory structure and address pair storage mechanism in the system, errors that may occur during the migration process can be effectively reduced, and the high accuracy of data migration is ensured.

[0074] In an optional embodiment, in order to further accurately store the address pair, step S203 comprises:

[0075] establishing a plurality of second directories, a plurality of second directories and at least one corresponding relationship of at least one cold data, and a plurality of second directories and a corresponding relationship of at least one first directory, wherein each second directory corresponds to one first directory and one cold data; according to the plurality of second directories and the at least one corresponding relationship, the address pair corresponding to each cold data is stored in the second directory corresponding to each cold data, and each second directory stores a corresponding address pair; obtaining the number of cold data in the migration task and the number of second directories used to store the address pair; if the number of cold data and the number of used directories are the same, according to the plurality of second directories and the corresponding relationship of the plurality of second directories and the at least one first directory, each address pair in each second directory is stored in each first directory.

[0076] In this embodiment, in order to avoid directly storing the address pair of the cold data in the first directory, the second directory is introduced in the system. Each second directory corresponds to one cold data and one first directory. According to the correspondence between the cold data and the second directory, the address pair of the cold data is stored in the corresponding second directory. When performing the migration task, the system will first obtain the number of cold data to be migrated in the migration task, and also obtain the number of second directories used to store the address pair. If the number of cold data and the number of second directories are the same, it indicates that the address pair of all cold data has been correctly stored in the second directory. When the number of cold data and the number of second directories are consistent, according to the mapping relationship between the second directory and the first directory, the corresponding address pair in each second directory is stored in the corresponding first directory.

[0077] The method provided in the embodiment avoids the address pair incompleteness caused by the direct reading of the address pair by the migration node in the process of storing the address pair in the first directory, improves the organization and management efficiency of the cold data migration, optimizes the utilization of the storage resources, reduces the storage pressure, realizes the automatic migration process, reduces the manual intervention, improves the scalability and flexibility of the system, and guarantees the accuracy and efficiency in the data migration process.

[0078] In an optional implementation, in order to further accurately store the address pair, step S203 comprises:

[0079] generating a plurality of first indexes, each of the first indexes being associated with an address pair; and storing each of the first indexes and each of the address pairs into each of the first directories.

[0080] In the embodiment, one first index is associated with one address pair. The first index is an index that cannot be recognized by the migration node, and the index can be a name identifier of the address pair. The manner of generating the plurality of first indexes can be a naming process for each address pair. The first index and the address pair can form a key-value pair, the first index serving as the key and the address pair serving as the value. One storage manner is to store each address pair with the name identifier in the form of the key-value pair into each corresponding first directory.

[0081] The method provided in the embodiment can quickly locate and access the cold data by associating the storage address of the cold data with the first index. The storage position of each cold data is accurately identified and indexed, avoiding the long search time that can occur in the traditional method. The directory is used to store the index and the address pair, so that the management of the cold data is more systematic and structured. The cold data usually has a low access frequency, but the use of the method can ensure that the cold data can be quickly obtained when needed.

[0082] In an optional implementation, in order to further accurately store the address pair, Figure 3 is a flowchart of a second cold data migration method according to an embodiment of the application, and step S203 comprises:

[0083] Step S301 acquires the number of cold data in the migration task and the number of used first directories in which the address pairs are stored.

[0084] The number of cold data is the total number of a plurality of tasks in the migration task, and the number of use is the number of the first directories in which the address pairs are stored.

[0085] One acquisition manner is to read the cold data quantity identifier carried in the migration task, to indicate the total quantity of cold data in the task, and to generate a use identifier after each first directory generates a storage address pair, to indicate that the first directory has been used, to count the quantity of first directories with the use identifier, and to reach the use quantity.

[0086] In step S302, if the cold data quantity and the use quantity are the same, each first index is modified to a corresponding second index in the first directory.

[0087] Each first index corresponds to a second index, each second index corresponds to a migration node, and each second index is used to indicate that each corresponding migration node acquires an associated address pair.

[0088] One modification manner is that, in the case where the first index is the name or key of the address pair, the name is modified to another name or key that can be recognized by another migration node, i.e., the second index. After the modification, the migration node can extract the address pair corresponding to the second index in the first directory, and then migrate the cold data.

[0089] The method provided in the embodiment optimizes the processing procedure of cold data migration by introducing the concepts of second index and migration node. When the cold data quantity matches the use quantity, the system converts the first index to the second index, thereby allocating the migration task of cold data to the migration node, and improving the efficiency, flexibility and scalability of the migration procedure. Meanwhile, the design of the migration node strengthens the integrity and consistency of data, and ensures efficient access and reliability in the cold data migration procedure.

[0090] In an optional implementation, in order to further monitor the migration procedure, the migration of the cold data corresponding to the address pair from the near-end cluster address to the destination address of the far-end cluster includes: migrating the cold data from the near-end cluster address to the destination address of the far-end cluster in batches according to splitting into multiple data objects; in the process of migrating the cold data from the near-end cluster address to the destination address of the far-end cluster in batches according to splitting into multiple data objects, further includes: if an update request or access time access is received, stopping the current migration operation of the cold data, and determining the data objects that have completed migration on the destination address of the far-end cluster; sending the data objects that have completed migration to the near-end cluster address, to combine the data objects that have completed migration with the data objects that have not been migrated, and then performing the update or access operation on the complete cold data.

[0091] In this embodiment, each cold data is not a single data item, but is composed of multiple data objects. These data objects can be sub-files, records, data blocks, etc. During the migration process, the last update time or the last access time of each cold data object is recorded to determine whether the data is active or should be migrated. If the update time or access time is greater than the migration time, it means that the cold data is modified or accessed during the migration process, so the object migrated to the remote cluster needs to be migrated back to the near-end cluster.

[0092] The method provided in this embodiment introduces data splitting, batch migration, interruption mechanism of update request, and recovery operation after migration completion, so that the system can more smoothly perform the migration task of cold data, ensure the high availability and consistency of cold data during the migration process, reduce the risk of data loss, and improve the fault tolerance of the system when facing high load and faults. These technical effects make the data migration more efficient, reliable, and better support large-scale and complex cold data migration tasks.

[0093] In an optional implementation, in order to further accurately migrate cold data, after the above step S203, the method further includes:

[0094] Each of the near-end cluster addresses and each of the remote cluster addresses is sent to the migration node mounted by using each of the above first directories; an extraction instruction sent by each of the migration nodes is received, and each of the corresponding cold data is sent to each of the migration nodes, so that each of the cold data is stored in the corresponding remote cluster address in the remote cluster.

[0095] In this embodiment, the migration node extracts the cold data to be migrated from the near-end cluster where each cold data is located. The migration node uses the near-end cluster address to locate and extract these data. The cold data can be stored in a certain specific path or directory, and the migration node extracts these cold data from the near-end cluster address according to the address. The migration node stores the extracted cold data in the specified target location in the remote cluster through network transmission, that is, the remote cluster address corresponding to the cold data.

[0096] The method provided in this embodiment migrates the cold data from the near-end cluster to the remote cluster by using the migration node according to the address pair, and ensures that the cold data is migrated to the correct target cluster through extraction and matching of the address pair, thereby improving the accuracy and consistency of data migration. Unnecessary storage of cold data is avoided, and the utilization of storage space and bandwidth resources is optimized.

[0097] In an optional implementation, in order to further determine the migration progress, each of the cold data includes multiple objects.

[0098] When migrating the cold data corresponding to the above-mentioned address pair from the near-end cluster address to the destination address of the far-end cluster, it also includes: obtaining the total number of objects of each of the above-mentioned cold data, and the number of migration objects in each of the above-mentioned cold data migrated to the above-mentioned far-end cluster; comparing the total number of objects of each of the above-mentioned cold data with the number of migration objects to obtain a second comparison result for each cold data; determining a second target cold data from a plurality of the above-mentioned cold data according to the second comparison result of each cold data, the total number of objects of the above-mentioned second target cold data being the same as the number of migration objects; generating a migration completion flag, the above-mentioned migration completion flag indicating that the migration task corresponding to the above-mentioned second target cold data has been completed.

[0099] In this embodiment, each cold data set contains multiple objects. The total number of objects in each cold data set is determined, and during the migration process, the number of objects successfully migrated is tracked. For each cold data set, the system compares the total number of objects with the number of objects to be migrated, obtains a second comparison result, and determines whether the total number of objects and the number of objects to be migrated are the same. If they are the same, the migration task for that cold data set (i.e., the second target cold data set) is considered complete.

[0100] The method provided in this embodiment can more accurately track the migration progress of each cold data during the migration process by obtaining the total number of objects and the number of migration objects for each cold data and comparing them. By clearly comparing the number of migration objects and the total number of objects, omissions or duplications during the migration process can be effectively avoided. By analyzing the second comparison result of each cold data, the priority and resource allocation of the migration task can be intelligently determined. By generating a migration completion identifier to indicate the completion of the migration task of the second target cold data, the system can efficiently mark the completion status of each migration task. By comparing the total number of objects and the number of migration objects, the system can accurately determine the progress of each cold data migration task and avoid unnecessary waste of resources.

[0101] In an optional implementation, to further ensure that the migrated cold data can be used normally, after generating the migration completion mark, the following steps are further included:

[0102] Obtain metadata corresponding to each of the above cold data, the metadata including: the storage address of the above cold data; update the metadata corresponding to each of the above cold data according to each of the above remote cluster addresses, the metadata being used to access or modify the cold data corresponding to the above read and write operations when processing read and write operations.

[0103] In the present embodiment, the metadata is an index for accessing or modifying the cold data, used to find the exact location of the cold data storage when the cold data needs to be accessed. During the migration of the cold data, the data is migrated from the original storage location, i.e. the near-end cluster address, to the far-end cluster or other storage system. The data location after migration needs to be updated in the metadata. For example, assuming that the storage address of a certain cold data is originally the path / storage / clusterl / data1 of the near-end cluster, and after migration the cold data is stored in / storage / cluster2 / data1 of the far-end cluster, the storage address in the metadata should be updated to / storage / cluster2 / data1, and the relevant information of the far-end cluster, such as the address, storage service identifier, etc., can be recorded.

[0104] The method provided by the present embodiment can ensure that the read and write operations after data migration can accurately access the data in the far-end cluster by obtaining the metadata corresponding to each cold data and updating the metadata according to the far-end cluster address. After updating the metadata, the cold data migration state can be reflected in real time, ensuring that each system component can perceive the migration progress of the cold data. By updating the metadata in a timely manner, the system can avoid invalid access caused by incorrect storage addresses or cluster addresses. Through accurate metadata management, the efficiency and transparency of the life cycle management of the cold data are improved.

[0105] In an optional implementation, the method further includes: during the migration of the cold data, logging each migration task, and the logging includes:

[0106] At the start of the migration task, the start time of the migration task, the migration node, the identifier of the cold data, and the migration target address are recorded. During the migration process, the progress of the migration of each cold data, the amount of data migrated, and the migration state are recorded in real time. After the migration task is completed, the end time of the migration task, whether the migration is successful, any abnormality occurring during the migration process, and the recovery measures taken are recorded. According to the log records, the performance of the migration task is analyzed to provide data support for the optimization of subsequent migration tasks.

[0107] The method provided by the embodiment can clearly understand the basic information of each migration task by recording the start time of the migration task, the migration node, the cold data identifier and the migration target address, and can ensure the traceability and manageability of the task. Real-time recording of the progress, the migrated data volume and the migration state can help to monitor the execution of the migration task in real time during the migration process, to discover potential problems in time, and to make necessary adjustments. Recording the end time of the migration, whether the migration is successful or not, the abnormal situation and the recovery measures taken can help to analyze whether the migration is successfully completed, and to provide a basis for cause analysis and processing when an abnormality occurs. Through analysis of the migration log, the overall performance of the migration task can be evaluated, including the migration efficiency, the success rate, potential bottlenecks and places that need to be improved.

[0108] The embodiment of the present application also provides a cold data migration device, Figure 4 is a schematic diagram of a cold data migration device according to the embodiment of the present application, the device is applied to a near-end cluster, and the device comprises:

[0109] The acquisition module 401 is used to acquire a migration task and a plurality of directories of a near-end cluster, the migration task comprises a plurality of address pairs, each address pair is associated with a cold data, each address pair comprises a near-end cluster address and a remote cluster address, the cold data is stored on the near-end cluster address, and the remote cluster address is a target address of the cold data to be migrated from the near-end cluster address.

[0110] The finding module 402 is used to find whether a first directory exists and whether the migration node is mounted on the first directory in the plurality of directories according to at least one cold data associated with the plurality of address pairs in the migration task.

[0111] The storage module 403 is used to store the address pair associated with the at least one cold data on at least one first directory corresponding to the at least one cold data, if the first directory is found, so that each migration node hung on the first directory migrates the cold data corresponding to the address pair from the near-end cluster address to the target address of the remote cluster when detecting that the address pair is stored in the first directory.

[0112] The present invention provides a cold data migration device, which can quickly find migration tasks related to cold data based on the correspondence between cold data and address pairs and store them in the corresponding first directory. By accurately searching the first directory containing the migration node, it can ensure that each cold data can get correct migration support. Each migration task corresponds to specific cold data and address pairs, making task management clearer and more organized. When at least one migration node obtains the address pair in the mounted first directory, the cold data migration task is automatically started. Manual intervention is reduced and the reliability and accuracy of the migration process are improved. By ensuring the smooth migration of data from the near-end cluster to the far-end cluster, the allocation of storage resources is optimized, which helps to reduce the storage pressure of cold data in the system. By using the directory structure and address pair storage mechanism in the system, it can effectively reduce the errors that may occur during the migration process and ensure the high accuracy of data migration.

[0113] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0114] The cold data migration device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0115] The embodiment of the present invention also provides a computer device having the above Figure 4 The migration device of cold data is shown.

[0116] See also Figure 5 , Figure 5 is a structural diagram of a computer device provided by an optional embodiment of the present invention, such as Figure 5 As shown, the computer device includes: one or more processors 10, memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components utilize different buses to communicate with each other and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in the memory or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Equally, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 10 is taken as an example.

[0117] The processor 10 can be a central processing unit, a network processing unit, or a combination thereof. The processor 10 can further include an integrated circuit. The integrated circuit can be an application specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device can be a complex programmable logic device, a field programmable logic device, a general array logic, or any combination thereof.

[0118] The memory 20 stores instructions executable by the at least one processor 10 for causing the at least one processor 10 to perform the methods illustrated by the embodiments.

[0119] The memory 20 can include a program storage area and a data storage area. The program storage area can store an operating system, application programs required by at least one function, and the like. The data storage area can store data created according to the use of the computer device, and the like. In addition, the memory 20 can include a high-speed random access memory, and can further include a non-transitory memory such as at least one of a magnetic disk storage device, a flash memory device, or other non-transitory solid state memory device. In some alternative embodiments, the memory 20 can optionally include a memory disposed remotely from the processor 10, and these remote memories can be connected to the computer device through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0120] The memory 20 can include a volatile memory such as a random access memory, and can also include a non-volatile memory such as a flash memory, a hard disk, or a solid state disk. The memory 20 can further include a combination of the above-mentioned types of memories.

[0121] The computer device further includes a communication interface 30 for communication of the computer device with other devices or communication networks.

[0122] The embodiments of the present application further provide a computer readable storage medium, and the method according to the embodiments of the present application can be implemented in hardware, firmware, or recorded in a storage medium, or stored in a remote storage medium or a non-transitory machine readable storage medium and downloaded to a local storage medium through network, so that the method described herein can be processed by such software on a storage medium using a general purpose computer, a special purpose processor, or programmable or special hardware. The storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid state disk, etc. Further, the storage medium can also include a combination of the above-mentioned memories. It can be understood that the computer, the processor, the microprocessor controller, or the programmable hardware includes a storage component that can store or receive software or computer code, when the software or computer code is accessed and executed by the computer, the processor, or the hardware, the method shown in the above embodiments is implemented.

[0123] Part of the present application can be applied as a computer program product, for example, computer program instructions, when executed by a computer, through the operation of the computer, the method and / or technical solutions according to the present application can be called or provided. Those skilled in the art should understand that the form of computer program instructions in a computer readable medium includes but is not limited to source files, executable files, installation package files, etc. Correspondingly, the way of executing computer program instructions by computer includes but is not limited to: the computer directly executes the instructions, or the computer compiles the instructions and then executes the corresponding compiled program, or the computer reads and executes the instructions, or the computer reads and installs the instructions and then executes the corresponding installed program. Here, the computer readable medium can be any available computer readable storage medium or communication medium accessible to the computer.

[0124] Although the embodiments of the present application are described in conjunction with the accompanying drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes fall within the scope defined by the appended claims.

Claims

1. A method for migrating cold data, characterized by, The method is applied to a near-end cluster, and the method comprises: obtaining a migration task and a plurality of directories of a near-end cluster, the migration task comprising a plurality of address pairs, each address pair being associated with one cold data, each address pair comprising a near-end cluster address and a remote cluster address, the cold data being stored on the near-end cluster address, and the remote cluster address being a destination address of the cold data to be migrated from the near-end cluster address; in accordance with at least one cold data associated with the plurality of address pairs in the migration task, searching for whether a first directory exists in the plurality of directories and whether the first directory is mounted with a migration node; if the first directory is found, storing the address pair associated with the at least one cold data on at least one first directory corresponding to the at least one cold data, so that each migration node mounted on the first directory migrates the cold data corresponding to the address pair from the near-end cluster address to the destination address of the remote cluster.

2. The method of claim 1, wherein, The storing of the address pair associated with the at least one cold data on the at least one first directory corresponding to the at least one cold data comprises: establishing a plurality of second directories, at least one correspondence between the plurality of second directories and at least one cold data, and a correspondence between the plurality of second directories and at least one first directory, wherein each second directory corresponds to one first directory and one cold data; in accordance with the plurality of second directories and the at least one correspondence, storing the address pair corresponding to each cold data in the second directory corresponding to each cold data, and each second directory stores one corresponding address pair; obtaining the number of cold data in the migration task and the number of second directories storing the address pair; if the number of cold data and the number of second directories are the same, storing each address pair in each first directory from each second directory in accordance with the plurality of second directories and the correspondence between the plurality of second directories and at least one first directory.

3. The method of claim 1, wherein, The storing of the address pair associated with the at least one cold data on the at least one first directory corresponding to the at least one cold data comprises: generating a plurality of first indexes, each first index being associated with one address pair; storing each first index and each address pair associated with each first index in each first directory.

4. The method of claim 3, wherein, After storing the address pair associated with the at least one cold data on the at least one first directory corresponding to the at least one cold data, the method further comprises: obtaining the number of cold data in the migration task and the number of first directories storing the address pair; if the number of cold data and the number of first directories are the same, modifying each first index to a corresponding second index in the first directory, each first index corresponding to one second index, each second index corresponding to one migration node, and each second index being used to instruct each corresponding migration node to obtain the associated address pair.

5. The method according to any one of claims 1 to 4, characterized in that, The method further comprises: migrating the cold data corresponding to the address pair from the near-cluster address to the destination address of the far cluster in batches according to the cold data being split into a plurality of data objects; In the process of migrating the cold data corresponding to the address pair from the near-cluster address to the destination address of the far cluster in batches according to the cold data being split into a plurality of data objects, the method further comprises: If an update request or an access time access is received, the current migration operation of the cold data is stopped, and it is determined that the data objects that have completed migration are on the destination address of the far cluster; The data objects that have completed migration are sent to the near-cluster address, and the update or access operation is performed on the complete cold data after the data objects that have completed migration are combined with the data objects that have not been migrated.

6. The method according to any one of claims 1 to 4, characterized in that, Each of the cold data comprises a plurality of objects; In the process of migrating the cold data corresponding to the address pair from the near-cluster address to the destination address of the far cluster, the method further comprises: obtaining the total number of objects of each of the cold data and the number of migrated objects in each of the cold data; comparing the total number of objects of each of the cold data with the number of migrated objects to obtain a second comparison result of each of the cold data; determining a second target cold data from the plurality of cold data according to the second comparison result of each of the cold data, the total number of objects of the second target cold data being the same as the number of migrated objects; generating a migration completion identifier, the migration completion identifier indicating that the migration task corresponding to the second target cold data is completed.

7. The method of claim 6, wherein, After the migration completion identifier is generated, the method further comprises: obtaining metadata corresponding to each of the cold data, the metadata comprising a storage address of the cold data; updating the metadata corresponding to each of the cold data according to each of the far-cluster addresses, the metadata being used to access or modify the cold data corresponding to a read-write operation when the read-write operation is processed.

8. A cold data migration apparatus, characterized by comprising: The device is applied to a near cluster, and the device comprises: an obtaining module, configured to obtain a migration task and a plurality of directories of a near cluster, the migration task comprising a plurality of address pairs, each of the address pairs being associated with a cold data, each of the address pairs comprising a near-cluster address and a far-cluster address, the cold data being stored on the near-cluster address, and the far-cluster address being a destination address of the cold data to be migrated from the near-cluster address; a searching module, configured to search, according to at least one cold data associated with a plurality of address pairs in the migration task, whether a first directory exists in the plurality of directories and whether a migration node is mounted on the first directory; a storage module, configured to, if the first directory is found, store the address pairs associated with the at least one cold data on at least one first directory corresponding to the at least one cold data, so that each migration node hung on the first directory migrates the cold data corresponding to the address pair from the near-cluster address to the destination address of the far cluster when detecting that the address pair is stored on the first directory.

9. A computer device, comprising: The method further comprises: A memory and a processor, which are connected in communication with each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the cold data migration method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing a computer to perform the cold data migration method of any one of claims 1 to 7.

Images