Cold data migration method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of servers, and discloses a cold data migration method and device, and the method comprises the steps: obtaining a migration task and a plurality of directories of a near-end cluster; according to at least one piece of cold data associated with multiple address pairs in the migration task, whether a first directory exists in the multiple directories or not is searched, and a migration node is mounted on the first directory; if the first directory is found to exist, storing the address pair associated with the at least one piece of cold data to at least one first directory corresponding to the at least one piece of cold data, so that when each migration node hung in the first directory detects that one address pair is stored in the first directory, the at least one piece of cold data is stored in the at least one first directory; and migrating the cold data corresponding to the address pair from the address of the near-end cluster to the destination address of the far-end cluster. Errors possibly occurring in the migration process can be effectively reduced, and high accuracy of data migration is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of servers, and in particular to a cold data migration method, device, computer equipment and storage medium. Background Art

[0002] At present, in the design of modern distributed storage systems, as the amount of data and business scale continue to expand, how to effectively reduce costs while ensuring performance has become an important challenge. Especially when processing massive unstructured data, the system's storage solution needs to be flexible and scalable to cope with the growing storage needs. Massive unstructured data generally includes documents, pictures, videos, logs and other files. These data often have no fixed structure, and the update frequency and access mode vary. Data that is accessed infrequently or has not been updated for a long time is called cold data. With the continuous increase in data volume, the reasonable management of cold data has become one of the key strategies for optimizing storage resources and improving storage efficiency. Summary of the invention

[0003] In view of this, the present invention provides a cold data migration method, apparatus, computer equipment, and storage medium to solve the problem of how to migrate cold data.

[0004] In a first aspect, the present invention provides a cold data migration method, which is applied to a near-end cluster and includes:

[0005] Obtain a migration task and multiple directories of a proximal cluster, the migration task including multiple address pairs, each of which is associated with a cold data, each of which includes a proximal cluster address and a remote cluster address, the cold data is stored on the proximal cluster address, and the remote cluster address is a destination address to be migrated from the proximal cluster address;

[0006] According to at least one cold data associated with the multiple address pairs in the migration task, searching among the multiple directories whether there is a first directory and a migration node is mounted on the first directory;

[0007] If the above-mentioned first directory is found to exist, the address pair associated with the above-mentioned at least one cold data is stored in at least one of the above-mentioned first directories corresponding to the above-mentioned at least one cold data, so that each migration node attached to the first directory will migrate the cold data corresponding to the above-mentioned address pair from the near-end cluster address to the destination address of the far-end cluster when it detects that an address pair is stored in the above-mentioned first directory.

[0008] The present invention provides a cold data migration method, which can quickly find migration tasks related to cold data based on the association relationship between cold data and address pairs and store them in the corresponding first directory. By accurately searching the first directory containing the migration node, it can ensure that each cold data can get correct migration support. Each migration task corresponds to specific cold data and address pairs, making task management clearer and more organized. When at least one migration node obtains the address pair in the mounted first directory, the cold data migration task is automatically started. Manual intervention is reduced and the reliability and accuracy of the migration process are improved. By ensuring the smooth migration of data from the near-end cluster to the far-end cluster, the allocation of storage resources is optimized, which helps to reduce the storage pressure of cold data in the system. By using the directory structure and address pair storage mechanism in the system, it is possible to effectively reduce errors that may occur during the migration process and ensure the high accuracy of data migration.

[0009] In an optional implementation manner, storing the address pair associated with the at least one cold data in the at least one first directory corresponding to the at least one cold data includes:

[0010] Establishing a plurality of second directories, at least one correspondence between the plurality of second directories and at least one cold data, and a correspondence between the plurality of second directories and at least one first directory, wherein each of the second directories corresponds to one first directory and one cold data;

[0011] According to the plurality of second directories and the at least one corresponding relationship, the address pair corresponding to each of the cold data is stored in the second directory corresponding to each of the cold data, and each of the second directories stores a corresponding address pair;

[0012] Obtaining the amount of cold data in the migration task and the number of used second directories storing the address pairs;

[0013] If the cold data quantity is the same as the used quantity, then according to the correspondence between the plurality of second directories and the plurality of second directories and at least one first directory, each of the address pairs is stored from each of the second directories to each of the first directories.

[0014] The method provided in this embodiment avoids incomplete address pairs caused by the migration node directly reading the address pairs in the process of storing address pairs in the first directory by introducing a multi-layer directory structure, an automated data migration process and precise address pair storage, thereby improving the organization and management efficiency of cold data migration, optimizing the utilization of storage resources, reducing storage pressure, realizing an automated migration process, reducing manual intervention, improving the scalability and flexibility of the system, and ensuring the accuracy and efficiency of the data migration process.

[0015] In an optional implementation manner, storing the address pair associated with the at least one cold data in the at least one first directory corresponding to the at least one cold data includes:

[0016] Generate a plurality of first indexes, each of which is associated with an address pair;

[0017] Each of the first indexes and each of the address pairs are stored in each of the first directories.

[0018] The method provided in this embodiment enables the system to quickly locate and access cold data by associating the storage address of cold data with the first index. The storage location of each cold data is accurately identified and indexed, avoiding the lengthy search time that may occur in traditional methods. Using a directory to store index and address pairs makes the management of cold data more systematic and structured. Cold data is usually accessed less frequently, but using this method can ensure that cold data can be quickly obtained when needed.

[0019] In an optional implementation manner, after storing the address pair associated with the at least one cold data in the at least one first directory corresponding to the at least one cold data, the method further includes:

[0020] Obtaining the amount of cold data in the migration task and the usage amount of the first directory storing the address pair;

[0021] If the cold data quantity is the same as the used quantity, then in the first directory, each of the first indexes is modified to a corresponding second index, each of the first indexes corresponds to one of the second indexes, each of the second indexes corresponds to one of the migration nodes, and each of the second indexes is used to indicate that each of the corresponding migration nodes obtains an associated address pair.

[0022] The method provided in this embodiment optimizes the cold data migration process by introducing the concepts of the second index and the migration node. When the amount of cold data matches the amount of use, the system converts the first index into the second index, thereby assigning the migration task of the cold data to the migration node, improving the efficiency, flexibility and scalability of the migration process. At the same time, the design of the migration node strengthens the integrity and consistency of the data, ensuring efficient access and reliability during the cold data migration process.

[0023] In an optional implementation manner, the above-mentioned migrating the cold data corresponding to the above-mentioned address pair from the local cluster address to the destination address of the remote cluster includes:

[0024] The cold data is split into multiple data objects and migrated in batches from the local cluster address to the destination address of the remote cluster;

[0025] In the process of splitting the cold data into multiple data objects and migrating them in batches from the local cluster address to the destination address of the remote cluster, the following steps are also included:

[0026] If an update request or access time access is received, the current cold data migration operation is stopped, and the data objects that have been migrated are determined at the destination address of the remote cluster;

[0027] The data objects that have been moved are sent to the near-end cluster address, so that the data objects that have been moved are combined with the data objects that have not been moved, and then an update or access operation is performed on the complete cold data.

[0028] The method provided in this embodiment, by introducing data splitting, batch migration, update request interruption mechanism and recovery operation after migration completion, can enable the system to perform cold data migration tasks more smoothly, ensure high availability and consistency of cold data during the migration process, reduce the risk of data loss, and improve the fault tolerance of the system in the face of high load and failures. These technical effects make data migration more efficient and reliable, and can better support large-scale and complex cold data migration tasks.

[0029] In an optional implementation, each of the above cold data includes multiple objects;

[0030] When migrating the cold data corresponding to the above address pair from the local cluster address to the destination address of the remote cluster, it also includes:

[0031] Obtain the total number of objects of each of the cold data and the number of objects migrated from each of the cold data to the remote cluster;

[0032] Compare the total number of objects of each of the cold data with the number of migrated objects to obtain a second comparison result of each of the cold data;

[0033] Determine second target cold data from the plurality of cold data according to the second comparison result of each cold data, wherein the total number of objects in the second target cold data is the same as the number of migration objects;

[0034] A migration completion mark is generated, where the migration completion mark indicates that the migration task corresponding to the second target cold data is completed.

[0035] The method provided in this embodiment can more accurately track the migration progress of each cold data in the migration process by obtaining the total number of objects and the number of migration objects of each cold data and comparing them. By clearly comparing the number of migration objects and the total number of objects, omissions or duplications in the migration process can be effectively avoided. By analyzing the second comparison result of each cold data, the priority and resource allocation of the migration task can be intelligently determined. By generating a migration completion mark to indicate that the migration task of the second target cold data is completed, the system can efficiently mark the completion status of each migration task. By comparing the total number of objects and the number of migration objects, the system can accurately determine the progress of each cold data migration task and avoid unnecessary waste of resources.

[0036] In an optional implementation, after the migration completion mark is generated, the process further includes:

[0037] Obtain metadata corresponding to each of the cold data, the metadata including: a storage address of the cold data;

[0038] According to each of the remote cluster addresses, metadata corresponding to each of the cold data is updated, where the metadata is used to access or modify the cold data corresponding to the read and write operations when processing the read and write operations.

[0039] The method provided in this embodiment can ensure that the read and write operations after data migration can accurately access the data in the remote cluster by obtaining the metadata corresponding to each cold data and updating the metadata according to the remote cluster address. After the metadata is updated, the cold data migration status can be reflected in real time to ensure that each system component can perceive the migration progress of the cold data. By updating the metadata in a timely manner, the system can avoid invalid access caused by incorrect storage addresses or cluster addresses. Through accurate metadata management, the efficiency and transparency of the life cycle management of cold data are improved.

[0040] In a second aspect, the present invention provides a cold data migration device, which is applied to a proximal cluster and includes:

[0041] An acquisition module is used to acquire a migration task and multiple directories of a proximal cluster, wherein the migration task includes multiple address pairs, each of which is associated with a cold data, each of which includes a proximal cluster address and a remote cluster address, the proximal cluster address stores the cold data, and the remote cluster address is a destination address to be migrated from the proximal cluster address;

[0042] A search module, configured to search, according to at least one cold data associated with the multiple address pairs in the migration task, whether there is a first directory in the multiple directories and a migration node is mounted on the first directory;

[0043] The storage module is used to store the address pair associated with the at least one cold data in at least one of the first directories corresponding to the at least one cold data if the first directory is found to exist, so that each migration node attached to the first directory can migrate the cold data corresponding to the address pair from the near-end cluster address to the destination address of the far-end cluster when detecting that an address pair is stored in the first directory.

[0044] The present invention provides a cold data migration device, which can quickly find migration tasks related to cold data based on the association relationship between cold data and address pairs and store them in the corresponding first directory. By accurately searching the first directory containing the migration node, it can ensure that each cold data can get correct migration support. Each migration task corresponds to specific cold data and address pairs, making task management clearer and more organized. When at least one migration node obtains the address pair in the mounted first directory, the cold data migration task is automatically started. Manual intervention is reduced and the reliability and accuracy of the migration process are improved. By ensuring the smooth migration of data from the near-end cluster to the far-end cluster, the allocation of storage resources is optimized, which helps to reduce the storage pressure of cold data in the system. By using the directory structure and address pair storage mechanism in the system, it can effectively reduce the errors that may occur during the migration process and ensure the high accuracy of data migration.

[0045] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the cold data migration method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0046] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the cold data migration method of the first aspect or any corresponding embodiment thereof.

[0047] In a fifth aspect, the present invention provides a computer program product, including computer instructions, which are used to enable a computer to execute the cold data migration method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 is a schematic diagram of a scenario architecture for cold data migration according to an embodiment of the present invention;

[0050] Figure 2 is one of the flow charts of the cold data migration method according to an embodiment of the present invention;

[0051] Figure 3 This is a second flowchart of the cold data migration method according to an embodiment of the present invention;

[0052] Figure 4 is a schematic diagram of a cold data migration device according to an embodiment of the present invention;

[0053] Figure 5 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0055] According to an embodiment of the present invention, an embodiment of a cold data migration method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0056] Figure 1 is a schematic diagram of a scenario architecture for cold data migration according to an embodiment of the present invention. Figure 1 As shown, the architecture includes: a near-end cluster, a far-end cluster and a client.

[0057] The near-end cluster includes a migration node, a file system, and a user node. Among them, the near-end cluster can be a storage system that stores and processes hot data with high frequency access and high real-time requirements, is geographically close to the user, has low network latency and high performance. The near-end cluster is connected to the client and the far-end cluster respectively. The file system includes multiple directories, for example, first directory 1, first directory 2, and first directory 3. The migration node is mounted under multiple first directories and is used to read the address pairs stored in the first directory. For example, migration node 1 is mounted under first directory 1, migration node 2 is mounted under first directory 2, and migration node 3 is mounted under first directory 3. The user node is used to connect to the client and receive the migration task sent by the user through the client. The migration node and the user node can be the same node, and the file system can also be deployed on the node. The far-end cluster is used to store the cold data of the near-end cluster.

[0058] In this embodiment, a cold data migration method is provided, which can be used for a near-end cluster. Figure 2 FIG. 1 is a flow chart of a cold data migration method according to an embodiment of the present invention. Figure 2 As shown, the method includes:

[0059] Step S201, obtaining migration tasks and multiple directories of the near-end cluster.

[0060] The migration task includes multiple address pairs, each address pair is associated with a cold data, each address pair includes a near-end cluster address and a far-end cluster address, the near-end cluster address stores the cold data, and the far-end cluster address is the destination address to be migrated from the near-end cluster address.

[0061] Among them, an address pair includes a corresponding address for storing cold data in the near-end cluster and a storage address to which the cold data will be migrated to the far-end cluster. The cold data in the near-end cluster or the far-end cluster can be stored in one address or in multiple addresses in segments, that is, one cold data can correspond to multiple near-end cluster addresses or far-end cluster addresses, and multiple addresses can be associated through the unique identity of the cold data.

[0062] One way of obtaining the data is to receive the migration task sent by the client through the user node connected to the client, and obtain multiple directories from the file system of the near-end cluster.

[0063] Step S202: searching among multiple directories for a first directory and a migration node mounted on the first directory according to at least one cold data associated with multiple address pairs in the migration task.

[0064] One cold data corresponds to one first directory. The multiple directories in the file system may be directories with or without mounted migration nodes, and may be directories with or without corresponding cold data.

[0065] One search method is to read the identification information of each directory, where the identification information is preset information indicating whether the directory is mounted with a migration node and corresponding cold data, or to search for the first directory corresponding to each cold data from multiple directories based on the correspondence between the cold data and the first directory.

[0066] Furthermore, if the first directory of the cold data is not found, a directory that does not correspond to any cold data can be selected from multiple directories as the first directory corresponding to the cold data. If no migration node is mounted under the first directory corresponding to the cold data, a migration node can be selected from the migration nodes that are not mounted under the directory and mounted to the first directory.

[0067] Step S203, if the first directory is found to exist, the address pair associated with at least one cold data is stored in at least one first directory corresponding to at least one cold data, so that each migration node attached to the first directory migrates the cold data corresponding to the address pair from the near-end cluster address to the destination address of the far-end cluster when detecting that an address pair is stored in the first directory.

[0068] A first directory stores a corresponding address pair of cold data. The correspondence between the cold data and the first directory can be preset, or can be a migration directory for each cold data specified by the user and carried in the migration task.

[0069] One storage method is: analyzing the correspondence between the cold data and the first directory, determining the first directory corresponding to each cold data, and storing the address pair corresponding to each cold data in the first directory corresponding to the cold data.

[0070] The address pair includes the address of the near-end cluster that currently stores the corresponding cold data and the address of the far-end cluster that will store the cold data.

[0071] One migration method is: the migration node reads the address pair in the first directory mounted by it, parses the local cluster address and the remote cluster address in the address pair, extracts the corresponding cold data from the local cluster address, and then stores the cold data in the corresponding remote cluster address in the remote cluster.

[0072] The above steps may be executed by one or more nodes of the proximal cluster, and the present invention is not limited here.

[0073] The present invention provides a cold data migration method, which can quickly find migration tasks related to cold data based on the correspondence between cold data and address pairs and store them in the corresponding first directory. By accurately searching the first directory containing the migration node, it can ensure that each cold data can get correct migration support. Each migration task corresponds to specific cold data and address pairs, making task management clearer and more organized. When at least one migration node obtains the address pair in the mounted first directory, the cold data migration task is automatically started. Manual intervention is reduced and the reliability and accuracy of the migration process are improved. By ensuring the smooth migration of data from the near-end cluster to the far-end cluster, the allocation of storage resources is optimized, which helps to reduce the storage pressure of cold data in the system. By using the directory structure and address pair storage mechanism in the system, it is possible to effectively reduce errors that may occur during the migration process and ensure the high accuracy of data migration.

[0074] In an optional implementation, in order to further accurately store the address pair, step S203 includes:

[0075] Establish a plurality of second directories, at least one correspondence between the plurality of second directories and at least one cold data, and a correspondence between the plurality of second directories and at least one first directory, wherein each second directory corresponds to a first directory and a cold data; store the address pair corresponding to each cold data in the second directory corresponding to each cold data according to the plurality of second directories and at least one correspondence, and each second directory stores a corresponding address pair; obtain the number of cold data in the migration task and the number of used data of the plurality of second directories storing the address pairs; compare and if the number of cold data is the same as the number of used data, then store each address pair from each second directory to each first directory according to the correspondence between the plurality of second directories and the plurality of second directories and at least one first directory.

[0076] In this embodiment, in order to avoid directly storing the address pairs of cold data in the first directory, a second directory is introduced into the system. Each second directory corresponds to a cold data and a first directory. According to the correspondence between the cold data and the second directory, the address pairs of the cold data are stored in its corresponding second directory. When executing a migration task, the system will first obtain the number of cold data that needs to be migrated in the migration task, and also obtain the number of used second directories storing address pairs. If the number of cold data and the number of used second directories are the same, it means that all the address pairs of cold data have been correctly stored in the second directory. When the number of cold data and the number of used second directories are consistent, according to the mapping relationship between the second directory and the first directory, the corresponding address pairs are stored in the corresponding first directory from each second directory.

[0077] The method provided in this embodiment avoids incomplete address pairs caused by the migration node directly reading the address pairs in the process of storing address pairs in the first directory by introducing a multi-layer directory structure, an automated data migration process and precise address pair storage, thereby improving the organization and management efficiency of cold data migration, optimizing the utilization of storage resources, reducing storage pressure, realizing an automated migration process, reducing manual intervention, improving the scalability and flexibility of the system, and ensuring the accuracy and efficiency of the data migration process.

[0078] In an optional implementation, in order to further accurately store the address pair, step S203 includes:

[0079] Generate multiple first indexes, each of which is associated with an address pair; store each of the first indexes and each of the address pairs in each of the first directories.

[0080] In this embodiment, a first index is associated with an address pair. The first index is an index that cannot be recognized by the migration node, and the index can be a name identifier of an address pair. The method of generating multiple first indexes can be a process of naming each address pair. Among them, the first index can form a key-value pair with the address pair, with the first index as the key and the address pair as the value. One storage method is: each address pair with a name identifier is stored in the form of a key-value pair in each corresponding first directory.

[0081] The method provided in this embodiment enables the system to quickly locate and access cold data by associating the storage address of cold data with the first index. The storage location of each cold data is accurately identified and indexed, avoiding the lengthy search time that may occur in traditional methods. Using a directory to store index and address pairs makes the management of cold data more systematic and structured. Cold data is usually accessed less frequently, but using this method can ensure that cold data can be quickly obtained when needed.

[0082] In an optional implementation, in order to further accurately store the address pair, Figure 3 This is a second flowchart of the cold data migration method according to an embodiment of the present invention, step S203 includes:

[0083] Step S301, obtaining the amount of cold data in the migration task and the usage amount of the first directory storing the address pairs.

[0084] The number of cold data is the total number of multiple tasks in the migration task, and the number of used data is the number of first directories that have stored address pairs.

[0085] One way to obtain it is: read the cold data quantity identifier carried in the migration task, indicating the total number of cold data in the task. The usage quantity can be a usage identifier generated after each first directory stores the address pair, indicating that the first directory has been used, and count the number of first directories with the usage identifier to reach the usage quantity.

[0086] Step S302: If the cold data quantity and the used data quantity are the same, in the first directory, each first index is modified to a corresponding second index.

[0087] Each first index corresponds to a second index, each second index corresponds to a migration node, and each second index is used to instruct each corresponding migration node to obtain an associated address pair.

[0088] One modification method is: when the first index is the name or key of the address pair, the name is modified to another name or key that the migration node can recognize, that is, the second index. After the modification, the migration node can extract the address pair corresponding to the second index in the first directory, and then migrate the cold data.

[0089] The method provided in this embodiment optimizes the cold data migration process by introducing the concepts of the second index and the migration node. When the amount of cold data matches the amount of use, the system converts the first index into the second index, thereby assigning the migration task of the cold data to the migration node, improving the efficiency, flexibility and scalability of the migration process. At the same time, the design of the migration node strengthens the integrity and consistency of the data, ensuring efficient access and reliability during the cold data migration process.

[0090] In an optional embodiment, in order to further monitor the migration process, the above-mentioned migration of the cold data corresponding to the above-mentioned address pair from the proximal cluster address to the destination address of the remote cluster includes: migrating the above-mentioned cold data from the proximal cluster address to the destination address of the remote cluster in batches according to splitting into multiple data objects; when the above-mentioned cold data is migrated from the proximal cluster address to the destination address of the remote cluster in batches according to splitting into multiple data objects, it also includes: if an update request or access time access is received, stopping the current migration operation on the cold data, and determining the data objects that have been moved at the destination address of the remote cluster; sending the data objects that have been moved to the proximal cluster address, so as to combine the data objects that have been moved with the data objects that have not yet been moved, and then perform an update or access operation on the complete cold data.

[0091] In this embodiment, each cold data is not a single data item, but is composed of multiple data objects. These data objects can be sub-files, records, data blocks, etc. During the migration process, the last update time or the last access time of each cold data object is recorded to determine whether the data is active or should be migrated. If the update time or access time is greater than the migration time, it means that the cold data has been modified or accessed during the migration process, so the objects migrated to the remote cluster need to be migrated back to the local cluster.

[0092] The method provided in this embodiment, by introducing data splitting, batch migration, update request interruption mechanism and recovery operation after migration completion, can enable the system to perform cold data migration tasks more smoothly, ensure high availability and consistency of cold data during the migration process, reduce the risk of data loss, and improve the fault tolerance of the system in the face of high load and failures. These technical effects make data migration more efficient and reliable, and can better support large-scale and complex cold data migration tasks.

[0093] In an optional implementation, in order to further accurately migrate cold data, after the above step S203, the following is further included:

[0094] Utilize each of the above-mentioned first directories to send each of the above-mentioned local cluster addresses and each of the above-mentioned remote cluster addresses to the mounted migration nodes; receive the extraction instructions sent by each of the above-mentioned migration nodes, and send each of the above-mentioned corresponding cold data to each of the above-mentioned migration nodes, so that each of the above-mentioned cold data is stored in the above-mentioned remote cluster address corresponding to the above-mentioned remote cluster.

[0095] In this embodiment, the migration node extracts the cold data to be migrated from the near-end cluster where each cold data is located. The migration node uses the near-end cluster address to locate and extract the data. The cold data may be stored in certain specific paths or directories, and the migration node extracts the cold data from the near-end cluster address according to the address. The migration node transmits the extracted cold data through the network and stores it in the target location specified in the far-end cluster, that is, the far-end cluster address corresponding to the cold data.

[0096] The method provided in this embodiment uses a migration node to migrate cold data from a near-end cluster to a far-end cluster according to address pairs, extracts and matches address pairs, ensures that cold data is migrated to the correct target cluster, and improves the accuracy and consistency of data migration. It avoids unnecessary storage of cold data and optimizes the use of storage space and bandwidth resources.

[0097] In an optional implementation, in order to further determine the migration progress, each of the above cold data includes multiple objects.

[0098] When the cold data corresponding to the above-mentioned address pair is migrated from the near-end cluster address to the destination address of the far-end cluster, it also includes: obtaining the total number of objects of each of the above-mentioned cold data, and the number of migration objects in each of the above-mentioned cold data migrated to the above-mentioned far-end cluster; comparing the total number of objects of each of the above-mentioned cold data with the number of migration objects to obtain a second comparison result of each cold data; determining a second target cold data from a plurality of the above-mentioned cold data according to the second comparison result of each cold data, the total number of objects of the above-mentioned second target cold data being the same as the number of migration objects; generating a migration completion mark, the above-mentioned migration completion mark indicating that the migration task corresponding to the above-mentioned second target cold data has been executed.

[0099] In this embodiment, each cold data contains multiple objects, the total number of objects in each cold data is determined, and during the migration process, the number of objects that have been successfully migrated is tracked. For each cold data, the system compares the total number of objects with the number of migrated objects, obtains a second comparison result, determines whether the total number of objects is the same as the number of migrated objects, and if they are the same, the migration task of the cold data (i.e., the second target cold data) is completed.

[0100] The method provided in this embodiment can more accurately track the migration progress of each cold data in the migration process by obtaining the total number of objects and the number of migration objects of each cold data and comparing them. By clearly comparing the number of migration objects and the total number of objects, omissions or duplications in the migration process can be effectively avoided. By analyzing the second comparison result of each cold data, the priority and resource allocation of the migration task can be intelligently determined. By generating a migration completion mark to indicate that the migration task of the second target cold data is completed, the system can efficiently mark the completion status of each migration task. By comparing the total number of objects and the number of migration objects, the system can accurately determine the progress of each cold data migration task and avoid unnecessary waste of resources.

[0101] In an optional implementation manner, in order to further ensure that the migrated cold data can be used normally, after the migration completion mark is generated, the following step is further included:

[0102] Obtain metadata corresponding to each of the above cold data, the metadata including: the storage address of the above cold data; update the metadata corresponding to each of the above cold data according to each of the above remote cluster addresses, the metadata is used to access or modify the cold data corresponding to the above read and write operations when processing read and write operations.

[0103] In this embodiment, metadata is an index used to access or modify cold data, and is used to find the exact location of the cold data storage when the cold data needs to be accessed. During the migration of cold data, the data is migrated from the original storage location, that is, the near-end cluster address, to the remote cluster or other storage system. The location of the migrated data needs to be updated in the metadata. For example, assuming that the original storage address of a certain cold data is the path / storage / cluster1 / data1 of the near-end cluster, and after migration, the cold data is stored in the remote cluster / storage / cluster2 / data1, then the storage address in the metadata should be updated to / storage / cluster2 / data1, and the relevant information of the remote cluster, such as the address, storage service identifier, etc., can be recorded.

[0104] The method provided in this embodiment can ensure that the read and write operations after data migration can accurately access the data in the remote cluster by obtaining the metadata corresponding to each cold data and updating the metadata according to the remote cluster address. After the metadata is updated, the cold data migration status can be reflected in real time to ensure that each system component can perceive the migration progress of the cold data. By updating the metadata in a timely manner, the system can avoid invalid access caused by incorrect storage addresses or cluster addresses. Through accurate metadata management, the efficiency and transparency of the life cycle management of cold data are improved.

[0105] In an optional implementation, the method further includes: during the cold data migration process, logging each migration task, where the log includes:

[0106] At the beginning of the migration task, record the start time of the migration task, the migration node, the cold data identifier, and the migration target address. During the migration process, record the progress of each cold data migration, the amount of data migrated, and the migration status in real time. After the migration task is completed, record the end time of the migration task, whether the migration is successful, any exceptions that occurred during the migration process, and the recovery measures taken. Analyze the performance of the migration task based on the log records to provide data support for the optimization of subsequent migration tasks.

[0107] The method provided in this embodiment can clearly understand the basic information of each migration task by recording the start time, migration node, cold data identifier and migration target address of the migration task, thereby ensuring the traceability and manageability of the task. Real-time recording of progress, amount of data migrated and migration status can help monitor the execution of the migration task in real time during the migration process, promptly identify potential problems and make necessary adjustments. Recording the end time of the migration, whether the migration is successful or not, abnormal situations that occur and recovery measures taken can help analyze whether the migration is completed smoothly and provide a basis for cause analysis and handling when abnormalities occur. By analyzing the migration log, the overall performance of the migration task can be evaluated, including migration efficiency, success rate, potential bottlenecks and areas for improvement.

[0108] The embodiment of the present invention also provides a cold data migration device. Figure 4 is a schematic diagram of a cold data migration device according to an embodiment of the present invention. The device is applied to a near-end cluster. The device includes:

[0109] The acquisition module 401 is used to obtain migration tasks and multiple directories of the proximal cluster, the above-mentioned migration tasks include multiple address pairs, each of the above-mentioned address pairs is associated with a cold data, each of the above-mentioned address pairs includes a proximal cluster address and a distal cluster address, the above-mentioned proximal cluster address stores the above-mentioned cold data, and the above-mentioned distal cluster address is the destination address of the above-mentioned cold data to be migrated from the proximal cluster address.

[0110] The search module 402 is used to search whether there is a first directory in the multiple directories and a migration node is mounted on the first directory according to at least one cold data associated with the multiple address pairs in the migration task.

[0111] The storage module 403 is used to store the address pair associated with the at least one cold data in at least one of the first directories corresponding to the at least one cold data if the first directory is found to exist, so that each migration node attached to the first directory can migrate the cold data corresponding to the address pair from the near-end cluster address to the destination address of the far-end cluster when it detects that an address pair is stored in the first directory.

[0112] The present invention provides a cold data migration device, which can quickly find migration tasks related to cold data based on the correspondence between cold data and address pairs and store them in the corresponding first directory. By accurately searching the first directory containing the migration node, it can ensure that each cold data can get correct migration support. Each migration task corresponds to specific cold data and address pairs, making task management clearer and more organized. When at least one migration node obtains the address pair in the mounted first directory, the cold data migration task is automatically started. Manual intervention is reduced and the reliability and accuracy of the migration process are improved. By ensuring the smooth migration of data from the near-end cluster to the far-end cluster, the allocation of storage resources is optimized, which helps to reduce the storage pressure of cold data in the system. By using the directory structure and address pair storage mechanism in the system, it can effectively reduce the errors that may occur during the migration process and ensure the high accuracy of data migration.

[0113] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0114] The cold data migration device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0115] The embodiment of the present invention also provides a computer device having the above Figure 4 The migration device of cold data is shown.

[0116] See also Figure 5 , Figure 5 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 5 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 10 is taken as an example.

[0117] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include an integrated circuit. The integrated circuit may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0118] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0119] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0120] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0121] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0122] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0123] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0124] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A cold data migration method, characterized in that: The method is applied to a proximal cluster, and the method comprises: Acquire a migration task and multiple directories of a proximal cluster, wherein the migration task includes multiple address pairs, each of which is associated with a cold data, and each of which includes a proximal cluster address and a remote cluster address, the cold data is stored on the proximal cluster address, and the remote cluster address is a destination address to be migrated from the proximal cluster address; According to at least one cold data associated with multiple address pairs in the migration task, searching among the multiple directories whether there is a first directory and a migration node is mounted on the first directory; If the first directory is found to exist, the address pair associated with the at least one cold data is stored in at least one of the first directories corresponding to the at least one cold data, so that each migration node attached to the first directory migrates the cold data corresponding to the address pair from the near-end cluster address to the destination address of the far-end cluster when detecting that an address pair is stored in the first directory.

2. The method according to claim 1, characterized in that: The storing the address pair associated with the at least one cold data in at least one first directory corresponding to the at least one cold data comprises: Establishing a plurality of second directories, at least one correspondence between the plurality of second directories and at least one cold data, and a correspondence between the plurality of second directories and at least one first directory, wherein each of the second directories corresponds to one first directory and one cold data; According to the multiple second directories and the at least one corresponding relationship, the address pair corresponding to each of the cold data is stored in the second directory corresponding to each of the cold data, and each of the second directories stores a corresponding address pair; Acquire the amount of cold data in the migration task and the usage amount of a plurality of second directories storing the address pairs; If the cold data quantity is the same as the used quantity, then according to the correspondence between the multiple second directories and the multiple second directories and at least one first directory, each of the address pairs is stored from each of the second directories to each of the first directories.

3. The method according to claim 1, characterized in that The storing the address pair associated with the at least one cold data in at least one first directory corresponding to the at least one cold data comprises: Generate a plurality of first indexes, each of which is associated with an address pair; Each of the first indexes and each of the associated address pairs are stored in each of the first directories.

4. The method according to claim 3, characterized in that After storing the address pair associated with the at least one cold data in at least one first directory corresponding to the at least one cold data, the method further includes: Acquire the amount of cold data in the migration task and the usage amount of the first directory storing the address pair; If the cold data quantity is the same as the used quantity, then in the first directory, each of the first indexes is modified to a corresponding second index, each of the first indexes corresponds to a second index, each of the second indexes corresponds to a migration node, and each of the second indexes is used to indicate that each corresponding migration node obtains an associated address pair.

5. The method according to any one of claims 1 to 4, characterized in that: The migrating the cold data corresponding to the address pair from the local cluster address to the destination address of the remote cluster includes: Migrating the cold data in batches from the local cluster address to the destination address of the remote cluster according to splitting it into multiple data objects; In the process of splitting the cold data into multiple data objects and migrating them in batches from the local cluster address to the destination address of the remote cluster, the process also includes: If an update request or access time access is received, the current cold data migration operation is stopped, and the data objects that have been migrated are determined at the destination address of the remote cluster; The data objects that have been moved are sent to the near-end cluster address, so that the data objects that have been moved are combined with the data objects that have not been moved, and then an update or access operation is performed on the complete cold data.

6. The method according to any one of claims 1 to 4, characterized in that: Each of the cold data includes a plurality of objects; When migrating the cold data corresponding to the address pair from the local cluster address to the destination address of the remote cluster, the method further includes: Obtaining the total number of objects of each of the cold data and the number of objects migrated from each of the cold data to the remote cluster; Compare the total number of objects of each cold data with the number of migration objects to obtain a second comparison result of each cold data; Determine second target cold data from the plurality of cold data according to the second comparison result of each cold data, wherein the total number of objects in the second target cold data is the same as the number of migration objects; A migration completion mark is generated, where the migration completion mark indicates that the migration task corresponding to the second target cold data is executed successfully.

7. The method according to claim 6, characterized in that After generating the migration completion mark, the method further includes: Obtain metadata corresponding to each of the cold data, the metadata comprising: a storage address of the cold data; According to each of the remote cluster addresses, metadata corresponding to each of the cold data is updated, where the metadata is used to access or modify the cold data corresponding to the read / write operation when processing the read / write operation.

8. A cold data migration device, characterized in that: The device is applied to a proximal cluster, and the device includes: An acquisition module is used to acquire a migration task and multiple directories of a near-end cluster, wherein the migration task includes multiple address pairs, each of which is associated with a cold data, and each of which includes a near-end cluster address and a far-end cluster address, wherein the near-end cluster address stores the cold data, and the far-end cluster address is a destination address to be migrated from the near-end cluster address; A search module, configured to search, according to at least one cold data associated with a plurality of address pairs in the migration task, whether a first directory exists in the plurality of directories and a migration node is mounted on the first directory; A storage module is used to store the address pair associated with the at least one cold data in at least one first directory corresponding to the at least one cold data if the first directory is found to exist, so that each migration node attached to the first directory can migrate the cold data corresponding to the address pair from the near-end cluster address to the destination address of the far-end cluster when detecting that an address pair is stored in the first directory.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the cold data migration method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the cold data migration method according to any one of claims 1 to 7.

Images