Data migration method and device, and computer readable storage medium

Abstract

The application provides a data migration method and device and a computer readable storage medium. The method comprises the following steps: receiving a data migration request, wherein the data migration request carries a data copy of to-be-migrated data; in response to the data migration request, configuring distributed storage resources and object storage resources for the data copy; and performing hybrid storage on the to-be-migrated data according to the distributed storage resources and the object storage resources. The distributed storage resources and the object storage resources are reasonably configured by using the method, and the hybrid storage resources can be used to solve the problem that diversified storage requirements cannot be met.

Description

Technical Field

[0001] This application relates to the field of big data processing technology, specifically to a data migration method, apparatus, and computer-readable storage medium. Background Technology

[0002] With the advent of the big data era, the amount of data in various enterprises has exploded, and most of them use distributed file systems such as HDFS to build storage systems.

[0003] However, the current data capacity stored in the distributed file system HDFS is becoming increasingly insufficient. Adding new machines requires a long period of load balancing, which can easily affect data read and write performance. While object storage can expand storage resources, effectively reducing costs without limiting capacity, it is still a challenge for enterprises with existing data in HDFS to migrate all data to object storage without downtime. Furthermore, it cannot meet the demand for elastic scaling of storage resources in a short period of time.

[0004] Therefore, existing data storage systems suffer from technical problems due to unreasonable allocation of storage resources, which prevents them from meeting diverse storage needs. Summary of the Invention

[0005] The purpose of this application is to provide a data migration method, apparatus, and computer-readable storage medium. By combining distributed storage and object storage, a hybrid storage scheme is designed to rationally configure distributed storage resources and object storage resources in the cloud. This solves the technical problems that cannot be met by the diverse storage needs of distributed file systems, such as data migration, storage capacity limitations, elastic expansion and contraction of storage resources, and reduction of storage resource costs.

[0006] Firstly, this application provides a data migration method applied in a cloud server, comprising:

[0007] Receive a data migration request, which carries a copy of the data to be migrated;

[0008] In response to data migration requests, configure distributed storage resources and object storage resources for the data replicas;

[0009] The data to be migrated is stored in a hybrid manner, based on distributed storage resources and object storage resources.

[0010] In some embodiments of this application, after configuring distributed storage resources and object storage resources for the data replica in response to a data migration request, the method further includes: in response to the distributed storage resources not meeting the data volume of the data replica, creating at least one node resource in the object storage resources as a first new storage resource of the distributed storage resources; and in response to the distributed storage resources containing the first new storage resources meeting the data volume, performing a mixed storage operation on the data to be migrated.

[0011] In some embodiments of this application, in response to the distributed storage resource not meeting the data volume of the data replica, at least one node resource is created in the object storage resource as a first new storage resource of the distributed storage resource, including: in response to the distributed storage resource not meeting the data volume of the data replica, at least one node resource is created in the object storage resource; the at least one node resource is allocated to the distributed storage resource as a first new storage resource; and a mapping relationship between the storage node resource and the distributed storage resource, the mapping relationship being used when the resource is scaled up or down.

[0012] In some embodiments of this application, the data migration method further includes: in response to the distributed storage resource containing the first newly added storage resource not meeting the data volume requirement, selecting a target distributed storage resource with excessive load; determining a target node resource in the target distributed storage resource based on the resource gap required to store the data volume in the distributed storage resource; transferring the target node resource to the distributed storage resource as a second newly added storage resource based on the checkpoint information of the target node resource; and performing a mixed storage operation on the data to be migrated in response to the distributed storage resource containing the first newly added storage resource and the second newly added storage resource meeting the data volume requirement.

[0013] In some embodiments of this application, the target node resource is transferred to the distributed storage resource as a second newly added storage resource based on the checkpoint information of the target node resource. This includes: generating checkpoint information based on the metadata information pre-stored in the target node resource; and transmitting the checkpoint information to the distributed storage resource through the object storage resource, so as to add the target node resource to the distributed storage resource as a second newly added storage resource.

[0014] In some embodiments of this application, the data to be migrated is mixed-stored according to distributed storage resources and object storage resources, including: storing a data copy in the distributed storage resources and obtaining the data storage result; in response to the data storage result being successful, asynchronously storing the data copy in the object storage resources, so as to perform mixed-store storage of the data to be migrated.

[0015] Secondly, this application provides a data migration method applied to a local server, comprising:

[0016] Receive data migration requests carrying the data directory to be migrated;

[0017] In response to a data migration request, determine the data type of the data to be migrated contained in the data directory to be migrated;

[0018] Obtain a copy of the data to be migrated based on the data type of the data to be migrated;

[0019] A copy of the data is sent to the cloud server. This copy is used by the cloud server to configure distributed storage resources and object storage resources for hybrid storage of the data to be migrated.

[0020] In some embodiments of this application, obtaining a data copy of the data to be migrated according to the data type of the data to be migrated includes: in response to the data type of the data to be migrated being existing data, obtaining a copy to be deleted as a data copy based on a first copy set of the data to be migrated directory pre-stored in the local server; and in response to the data type of the data to be migrated being new data, obtaining a data copy based on a second copy set of the data to be migrated directory pre-stored in the local server.

[0021] Thirdly, this application provides a data migration apparatus, comprising:

[0022] The request receiving module is used to receive data migration requests, which carry a copy of the data to be migrated.

[0023] The resource configuration module is used to configure distributed storage resources and object storage resources for data replicas in response to data migration requests;

[0024] The data migration module is used to perform hybrid storage of the data to be migrated based on distributed storage resources and object storage resources.

[0025] Fourthly, this application provides a data migration apparatus, comprising:

[0026] The request receiving module is used to receive data migration requests carrying the data directory to be migrated;

[0027] The request and response module is used to respond to data migration requests and determine the data type of the data to be migrated contained in the data directory to be migrated.

[0028] The copy acquisition module is used to obtain a data copy of the data to be migrated based on the data type of the data to be migrated;

[0029] The copy sending module is used to send data copies to the cloud server. The data copies are used by the cloud server to configure distributed storage resources and object storage resources for hybrid storage of the data to be migrated.

[0030] Fifthly, this application also provides a computer device, comprising:

[0031] One or more processors;

[0032] The memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by a processor to implement the data migration method of the first or second aspect described above.

[0033] Sixthly, this application also provides a computer-readable storage medium having a computer program stored thereon, the computer program being loaded by a processor to perform the steps in the data migration method.

[0034] In a seventh aspect, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the first or second aspect described above.

[0035] The aforementioned data migration method and apparatus involve the cloud server receiving and responding to data migration requests from the local server, configuring distributed storage resources and object storage resources for the data replicas, and then performing hybrid storage of the data to be migrated based on these resources. This not only enables uninterrupted data migration through interaction with the cloud server, but also facilitates cross-data center disaster recovery backup to enhance data security, and achieves rapid load balancing of storage resources, reducing data storage costs and meeting diverse storage needs. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a schematic diagram illustrating a scenario of the data migration method provided in the embodiments of this application;

[0038] Figure 2 This is a schematic diagram of the hybrid storage system provided in the embodiments of this application;

[0039] Figure 3 This is a flowchart illustrating the data migration method provided in the embodiments of this application;

[0040] Figure 4 This is a timing diagram of the data migration method provided in the embodiments of this application;

[0041] Figure 5 This is a flowchart illustrating another data migration method provided in the embodiments of this application;

[0042] Figure 6 This is a schematic diagram of the data migration device provided in the embodiments of this application;

[0043] Figure 7 This is a schematic diagram of another data migration device provided in the embodiments of this application;

[0044] Figure 8 This is a schematic diagram of the structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0045] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0046] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0047] In the description of this application, the term "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0048] This application provides a data migration method, apparatus, and computer-readable storage medium, which will be described in detail below.

[0049] See Figure 1 , Figure 1 This is a schematic diagram of a scenario for the data migration method provided in an embodiment of this application. The data migration method can be applied to a hybrid storage system, which includes a local server 101 and a cloud server 102 connected via a wireless network. The cloud server 102 can be any one of a public cloud server, a private cloud server, or a hybrid cloud server.

[0050] See Figure 2 The local server 101 is equipped with a DataNode component, and the cloud server 102 is equipped with a CloudDataNode component (hereinafter referred to as CDN). The CloudDataNode component adds the functions of uploading and downloading blocks from OBS, local data block caching, checkpointing, and communication with the Coordinator component on the basis of the DataNode component.

[0051] In addition, a NameNode component is set up in the local server 101, which can communicate and interact with the terminal based on a preset communication connection; an OBS (Object Storage) component, a Coordinator component, and a RocksDB component are set up in the cloud server 102; the Coordinator component is used to quickly balance blocks and elastically scale the CDN to avoid hot spot problems of CDN nodes; the RocksDB component is used to store CDN status information and the mapping relationship between buckets and CDN to avoid loss of status information when the Coordinator component is restarted.

[0052] Typically, data blocks on a DataNode are stored on disk as files, consisting of two files: one containing the data itself, and the other containing data block metadata, including length, checksum, and timestamp. After starting, a DataNode registers with the NameNode service and periodically reports all data block metadata information to the NameNode. There is a heartbeat mechanism between the DataNode and the NameNode, for example, every 3 seconds, and the returned result contains the execution command from the NameNode to the DataNode, such as data replication or deletion. If no heartbeat is received from the DataNode within 10 minutes or other limited time, the node is considered unavailable.

[0053] More specifically, the terminal can be a desktop terminal or a mobile terminal, and can also be a mobile phone, tablet computer, or laptop computer. Local server 101 and cloud server 102 can be independent servers, or they can be a server network or server cluster integrated into a central server, including but not limited to computers, network hosts, single network servers, multiple network server sets, or cloud servers composed of multiple servers. The cloud server consists of a large number of computers or network servers based on cloud computing; the network includes, but is not limited to, wide area networks (WANs), metropolitan area networks (MANs), or local area networks (LANs).

[0054] It should be noted that those skilled in the art should understand that Figure 1 The application scenarios shown are merely one applicable scenario for the solution in this application and do not constitute a limitation on the application scenarios of the solution in this application. Other application environments may include more than one. Figure 1 The number of computer devices shown may be more or less. For example, Figure 1 Only two servers are shown in the illustration. It is understood that this hybrid storage system may also include one or more other servers or one or more other terminals, which are not specifically limited here. Those skilled in the art will recognize that, with the evolution of hybrid storage systems and the emergence of new business scenarios, the technical solutions provided in the embodiments of this invention are also applicable to similar technical problems.

[0055] See Figure 3 This application provides a data migration method, which will be mainly applied to the above-mentioned methods in the following text. Figure 1 Taking cloud server 102 as an example, the method includes steps S301 to S303, as follows:

[0056] S301, Receive a data migration request, which carries a copy of the data to be migrated.

[0057] In practical implementation, in order to solve the problem of not being able to meet diversified storage needs, such as the problem of migrating existing data without interruption, this application proposes to deploy distributed storage (Hadoop Distributed File System, HDFS) and object storage (OBS) in cloud servers, so as to take advantage of the advantages of the two storage resources, rather than relying on a single local storage method, to solve the problem of migrating all data stored in HDFS to OBS.

[0058] Therefore, when cloud server 102 receives a data migration request from local server 101, it can utilize pre-configured HDFS and OBS to respond to the request and migrate the specified data. Before this, cloud server 102 will parse the information in the data migration request, including but not limited to data copies. It should be noted that this data copy is actually a copy of the data to be migrated, not the data itself. The reason for migrating the data copy instead of the data itself is not only for cross-datacenter disaster recovery backup and improving the security of stored data, but also to facilitate efficiency improvements in actual business operations.

[0059] As is understandable, disaster recovery backup refers to establishing two or more IT systems with the same functions in remote locations. These systems can monitor each other's health status and switch functions. When one system stops working due to an accident (such as a fire or earthquake), the entire application system can be switched to the other system so that the system can continue to function normally.

[0060] S302, in response to a data migration request, configures distributed storage resources and object storage resources for the data replica.

[0061] Distributed storage resources can provide storage space for HDFS, including storage size and storage IP address. HDFS is a highly fault-tolerant system that can provide high-throughput data access, making it very suitable for applications on large-scale datasets.

[0062] Among them, object storage resources can provide storage space for OBS, including storage size, storage IP address, etc. OBS (Object Based Storage) is a cloud storage service based on a large-scale distributed, high-concurrency storage framework. It can store a large amount of unstructured data of any size and format, such as video, audio, documents, images, web page content, etc. Therefore, it can be widely used in scenarios such as content storage and distribution, big data analysis, data archiving and disaster recovery backup.

[0063] In the specific implementation, the cloud server 102 should be pre-configured with a processor. After the processor obtains the data copy of the data to be migrated, it will schedule the CloudDataNode component and the OBS component to configure distributed storage resources and object storage resources for the data copy, respectively.

[0064] In one embodiment, after this step, the data migration method further includes: in response to the distributed storage resource not meeting the data volume of the data replica, creating at least one node resource in the object storage resource as a first new storage resource of the distributed storage resource; and in response to the distributed storage resource containing the first new storage resource meeting the data volume, performing a mixed storage operation on the data to be migrated.

[0065] In the specific implementation, after the cloud server 102 configures distributed storage resources and object storage resources for the data replicas of the data to be migrated, if it is detected that the distributed storage resources are still insufficient to meet the data volume required by the current data replicas, the distributed storage resources can be supplemented. The resource supplementation method can be to create a new bucket in the object storage resource OBS, and then allocate the bucket to the CloudDataNode (CDN) component. After receiving the bucket, the CDN will use the bucket to upload or download blocks, thereby completing the elastic expansion of CDN nodes, that is, realizing the supplementation of distributed storage resources. The "bucket" created in OBS and transferred to CDN is the first newly added storage resource.

[0066] Furthermore, if the newly added distributed storage resources already meet the required data volume, then based on the distributed storage resources and object storage resources, a mixed storage operation of multiple resource types can be performed on the data to be migrated, that is, the data to be migrated can be stored in the distributed storage resources provided by the cloud server 102, and can also be stored in the object storage resources provided by the cloud server at the same time.

[0067] Of course, those skilled in the art can also perform hybrid storage operations in other ways. For example, in response to the distributed storage resources not meeting the data volume of the data replica, object storage resources can be used to store the data replica. Alternatively, in response to the distributed storage resources not meeting the data volume of the data replica, it can be detected whether the object storage resources meet the data volume of the data replica. If the object storage resources meet the data volume, it can be directly stored in OBS. Furthermore, it should be noted that the first newly added storage resource can be combined with the existing distributed storage resources to store the data replica. That is, the total resources of the first newly added storage resource and the existing distributed storage resources meet the data volume, and the first newly added storage resource can be used to store data volumes that the distributed storage resources cannot meet.

[0068] In one embodiment, in response to the distributed storage resource not meeting the data volume of the data replica, at least one node resource is created in the object storage resource as a first new storage resource of the distributed storage resource, including: in response to the distributed storage resource not meeting the data volume of the data replica, at least one node resource is created in the object storage resource; the at least one node resource is allocated to the distributed storage resource as a first new storage resource; and a mapping relationship between the storage node resource and the distributed storage resource, the mapping relationship being used when the resource is scaled up or down.

[0069] In specific implementation, based on the above embodiments, the step of supplementing distributed storage resources on cloud server 102 actually involves a CDN component applying for registration with the Coordinator component. After receiving the registration request, the Coordinator responds by creating a new bucket in OBS, then allocating the relevant information of the bucket to the CDN component (equivalent to data block transfer), and storing the mapping relationship between node resource buckets and distributed storage resources CDN in the RocksDB component to avoid intractable problems such as information loss when restarting the Coordinator. After receiving the relevant information of the bucket, the CDN component obtains the first newly added storage resource, and then uses the bucket to upload or download blocks, thereby expanding the CDN nodes to supplement the distributed storage resources.

[0070] In one embodiment, the data migration method further includes: in response to the distributed storage resource containing the first newly added storage resource not meeting the data volume requirement, selecting a target distributed storage resource with excessive load; determining a target node resource in the target distributed storage resource based on the resource gap required to store the data volume in the distributed storage resource; transferring the target node resource to the distributed storage resource as a second newly added storage resource based on the checkpoint information of the target node resource; and performing a mixed storage operation on the data to be migrated in response to the distributed storage resource containing the first newly added storage resource and the second newly added storage resource meeting the data volume requirement.

[0071] The target distributed storage resource can be a CDN with a large number of buckets and blocks and excessive load.

[0072] In specific implementation, the above embodiments have explained that when the cloud server 102 detects that the distributed storage resources are insufficient to meet the data volume required for the current data replica, the distributed storage resources can be supplemented by expanding CDN nodes. This embodiment will provide another resource supplementation scheme, namely, realizing CDN load balancing, which can also improve system read and write performance and reduce system maintenance costs. It should be noted that the other resource supplementation scheme provided in this embodiment actually supplements the situation where the first newly added storage resource is still insufficient to meet the data volume due to data volume estimation errors or misallocation of the first newly added storage resource in the previous steps. For example, the actual data volume of the data replica is "5", but due to an evaluation error in the previous steps, the data volume is mistakenly evaluated as "4". With the distributed storage resource itself being "3", the obtained first newly added storage resource is "1". At this time, the distributed storage resource "3" containing the first newly added storage resource "1" is insufficient to meet the data volume "5". Therefore, a second newly added storage resource "1" is added so that all resources together: 1+3+1=5, which can meet the data volume "5".

[0073] Specifically, before acquiring the second new storage resource, the Coordinator component selects a suitable source CDN for scaling down based on the number of buckets and blocks in each CDN. The source CDN may be overloaded, as it may have many buckets that are not fully utilized, leading to excessive load. Therefore, this embodiment proposes scaling down the source CDN when the available storage space of the distributed storage resource is insufficient. The actual scaling down size is based on the resource gap, which is the difference between the amount of data required to store data replicas and the distributed storage resource containing the first new storage resource. After determining the resource gap size, the number of nodes to be transferred to fill the gap can be calculated based on the available storage space of each candidate node resource. This determines the target node resource in the target distributed storage resource. Then, based on the checkpoint information of the target node resource, the target node resource is transferred to the distributed storage resource to obtain the second new storage resource, thereby supplementing the distributed storage resource by transferring CDN nodes.

[0074] For example, see Figure 4The target distributed storage resource is "CDN1". The distributed storage resource "CDN2" is a portion that does not meet the required data volume but contains the first newly added storage resource "bucket1". Cloud server 102 can then transfer one or more target node resources "bucket2" from "CDN1" to the distributed storage resource "CDN2" based on the resource gap in "CDN2". Specifically, node resource transfer can be performed using checkpoint information. At this point, the transferred "bucket2" becomes the second newly added storage resource. Both the second and first newly added storage resources can be used as new storage resources for "CDN2".

[0075] It should be noted that the storage resource transfers mentioned in the embodiments of this application all require the use of the metadata of the data block. For example, the cloud server 102 generates a checkpoint information "checkpoint" based on the metadata of the data block "bucket2" to be transferred. After the target distributed storage resource "CDN1" uploads the "checkpoint", the distributed storage resource "CDN2" downloads the "checkpoint", thus realizing the transfer of storage resources from one CDN to another.

[0076] In one embodiment, transferring the target node resource to a distributed storage resource as a second newly added storage resource based on the checkpoint information of the target node resource includes: generating checkpoint information based on metadata information pre-stored in the target node resource; and transmitting the checkpoint information to the distributed storage resource through an object storage resource, so as to add the target node resource to the distributed storage resource as a second newly added storage resource.

[0077] Metadata information includes, but is not limited to, the length, validation, and timestamp of the data itself.

[0078] In the specific implementation, the acquisition of the second newly added storage resource is controlled by the Coordinator component. The Coordinator component first notifies the target distributed storage resource (also known as the source CDN) to release the bucket. After receiving the message, the source CDN will generate checkpoint information (also known as checkpoint) for the metadata information of all blocks under the bucket, and then asynchronously upload it to the object storage OBS component. After that, the Coordinator component informs the distributed storage resource to be supplemented to add the bucket of the source CDN, download the checkpoint information from OBS, and load the metadata information of all blocks under the bucket, so that the distributed storage resource can be supplemented.

[0079] For example, see Figure 4Resource replenishment tasks for the distributed storage resource "CDN2" can be triggered by the client, but it is not required that the client initiate them. The client (cli) can send a "scaleDown num" request to the Coordinator, requesting the Coordinator in the hybrid storage system to select a suitable target distributed storage resource to release bucket and / or block node resources for the distributed storage resource "CDN2" that is experiencing storage shortages to replenish resources. For example, in response to the client's scaling-down request, the Coordinator determines "CDN1" as the target distributed storage resource (it can be understood that, in conjunction with the embodiments described above, the unused resources contained in "CDN1" should be greater than or equal to the resource gap size. If there are multiple CDNs that can be selected as target distributed storage resources, they can be sorted in descending order according to the unused resources of each CDN, and then the first CDN in the sequence is taken as the target distributed storage resource, such as "CDN1" mentioned in this embodiment). Based on the analyzed resource gap size, "bucket1" and "bucket2" in "CDN1" are further determined as target node resources. These two target node resources can be moved to "CDN2" to release storage resources for "CDN2".

[0080] Furthermore, after the Coordinator analyzes and determines the target node resources, it first needs to modify the node resource status of the target distributed storage resource "CDN1", that is, send the status modification command "BucketnactiveCommand" to "CDN1" to change it to the "nactive" status. After "CDN1" sends back the status modification confirmation information (ack), the Coordinator continues to send the node resource deletion command "BucketDeleteCommand" to "CDN1" to notify "CDN1" to release "bucket1" and "bucket2".

[0081] Furthermore, after "CND1" sends back confirmation of node resource release, the Coordinator will instruct "CND1" to send a node resource movement report to the "NameNode" via the command "Data NodeBlockReportCommand". If "CND1" lacks sufficient node resources, it will perform a data block write operation. Upon receiving the message, "CND1" will generate a checkpoint for the metadata information of node resources "bucket1" and "bucket2" and then asynchronously upload it to OBS.

[0082] Furthermore, after completing the above operations, the Coordinator can use "BucketAddCommand" to inform "CDN2" to add "bucket1" and "bucket2". After changing the node resource status of "bucket1" and "bucket2" from "nactive" to "active", it downloads the checkpoints of "bucket1" and "bucket2" from OBS, loads the metadata information of all blocks under "bucket1" and "bucket2", and finally sends a node resource movement report to "NameNode", thus realizing the resource supplementation of "CND2".

[0083] S303 performs hybrid storage of the data to be migrated based on distributed storage resources and object storage resources.

[0084] In specific implementations, for example, this embodiment proposes to use hybrid storage resources to migrate existing data in HDFS to the public cloud without downtime. The number of data replicas migrated to the cloud server 102 can be one or more. In this way, the data to be migrated is still stored locally, and one replica is stored in the cloud. This not only serves the function of cross-data center disaster recovery and backup, but also frees up a large amount of storage space in the local data center, solving the problem of seamless data migration across systems.

[0085] For example, in one embodiment, this step includes: storing a data copy in a distributed storage resource and obtaining the data storage result; in response to the data storage result being successful, asynchronously storing the data copy in an object storage resource for hybrid storage of the data to be migrated.

[0086] In the data migration method described in the above embodiments, the cloud server receives and responds to data migration requests from the local server, configures distributed storage resources and object storage resources for the data replica, and performs hybrid storage of the data to be migrated based on the distributed storage resources and object storage resources. This not only enables uninterrupted data migration through interaction with the cloud server, but also allows for cross-data center disaster recovery backup to improve data security, and achieves rapid load balancing of storage resources, reducing data storage costs and meeting diverse storage needs.

[0087] See Figure 5 This application also provides another data migration method, which will be mainly applied to the above in the following description. Figure 1 Taking local server 101 as an example, the method includes steps S501 to S504, as follows:

[0088] S501, Receive a data migration request carrying a data directory to be migrated.

[0089] The data directory to be migrated can be the data directory of the data to be migrated, which includes buckets and blocks.

[0090] In specific implementation, the data migration request can be a request sent by a terminal, which can be a client. The client can respond to user-triggered instructions to obtain the data directory to be migrated, such as receiving a data directory to be migrated submitted by the user, or retrieving a pre-stored data directory to be migrated according to user instructions. Furthermore, after obtaining the data directory to be migrated, the client can generate a data migration request based on the target, and then send the data migration request carrying the data directory to be migrated to the local server 101, so that the local server 101 can receive the data migration request carrying the data directory to be migrated.

[0091] S502, in response to the data migration request, determines the data type of the data to be migrated contained in the data to be migrated directory.

[0092] In the specific implementation, after the local server 101 receives the data migration request, it can respond to the request by first determining the data type of the data to be migrated, so as to take different data migration methods according to the data type. This will be explained in detail below.

[0093] S503: Obtain a copy of the data to be migrated based on the data type of the data to be migrated.

[0094] In practice, data types include existing data and newly added data. The differences in data migration between existing data and newly added data are mainly reflected in the operation process of local server 101. Among them, newly added data (also known as incremental data) is data newly added to the database, while existing data is data that already exists in the database or can be called historical data. The sum of existing data and newly added data equals the total data.

[0095] For example, in one embodiment, this step includes: in response to the data type of the data to be migrated being existing data, obtaining a copy to be deleted as a data copy based on a first copy set of the data to be migrated directory pre-stored on the local server; and in response to the data type of the data to be migrated being new data, obtaining a data copy based on a second copy set of the data to be migrated directory pre-stored on the local server.

[0096] A replica set can be a collection of files containing at least one copy.

[0097] In specific implementation, local server 101 can delete and copy the first replica set of the data directory to be migrated if the data type to be migrated is existing data. That is, for existing data, since it already exists, local server 101 can delete one replica on the existing local replica and copy one replica to cloud server 102 after the user configures the cross-data center data directory, so as to reduce local resource waste while meeting disaster recovery backup requirements. In addition, local server 101 can also directly copy the second replica set of the data directory to be migrated if the data type to be migrated is new data. That is, for new data, since it did not exist before, if the data is contained in the cross-data center data directory, assuming that the number of replicas configured for the data directory is N (N≥2), local server 101 can store N-1 replicas locally and then copy one replica to cloud server 102. In other words, the data migration of existing data and new data always ensures that cloud server 102 retains only one replica of the data, realizing cross-data center disaster recovery backup.

[0098] S504 sends a data copy to the cloud server. The data copy is used by the cloud server to configure distributed storage resources and object storage resources for hybrid storage of the data to be migrated.

[0099] In practice, after the local server 101 obtains a copy of the data to be migrated, it can send the copy of the data to the cloud server 102 via a wireless network (such as 4G, 5G, etc.) to achieve efficient and seamless data migration.

[0100] In the data migration method described in the above embodiments, the local server receives and responds to a data migration request carrying a directory of data to be migrated, determines the data type of the data to be migrated contained in the directory, and then obtains a data copy of the data to be migrated based on the data type. This data copy is then sent to the cloud server for the cloud server to configure distributed storage resources and object storage resources, achieving hybrid storage of the data to be migrated. In this way, not only can data migration be achieved without downtime through interaction with the cloud server, but data can also be backed up across data centers to improve data security, meeting diverse storage needs.

[0101] To better implement the data migration method provided in the embodiments of this application, based on the data migration method proposed in the embodiments of this application, this application also provides a data migration apparatus, such as... Figure 6 As shown, the data migration device 600 includes:

[0102] The request receiving module 610 is used to receive data migration requests, which carry a copy of the data to be migrated.

[0103] Resource configuration module 620 is used to configure distributed storage resources and object storage resources for data replicas in response to data migration requests;

[0104] The data migration module 630 is used to perform hybrid storage of the data to be migrated based on distributed storage resources and object storage resources.

[0105] In one embodiment, the data migration apparatus 600 further includes a resource expansion module, configured to create at least one node resource in the object storage resource as a first new storage resource in response to the distributed storage resource not meeting the data volume of the data replica; and to perform a mixed storage operation on the data to be migrated in response to the distributed storage resource containing the first new storage resource meeting the data volume.

[0106] In one embodiment, the resource expansion module is further configured to, in response to the distributed storage resource not meeting the data volume of the data replica, create at least one node resource in the object storage resource; allocate the at least one node resource to the distributed storage resource as a first newly added storage resource; and establish a mapping relationship between the storage node resource and the distributed storage resource, the mapping relationship being used during resource expansion and contraction.

[0107] In one embodiment, the resource expansion module is further configured to: respond to the distributed storage resource containing the first newly added storage resource not meeting the data volume requirement, filter out target distributed storage resources with excessive load; determine the target node resource in the target distributed storage resource based on the resource gap required to store the data volume in the distributed storage resource; transfer the target node resource to the distributed storage resource as the second newly added storage resource based on the checkpoint information of the target node resource; and perform a mixed storage operation on the data to be migrated in response to the distributed storage resource containing the first newly added storage resource and the second newly added storage resource meeting the data volume requirement.

[0108] In one embodiment, the resource expansion module is further configured to generate checkpoint information based on metadata information pre-stored in the target node resource; and transmit the checkpoint information to the distributed storage resource through the object storage resource so as to add the target node resource in the distributed storage resource as a second new storage resource.

[0109] In one embodiment, the data migration module 630 is further configured to store a data copy in a distributed storage resource and obtain the data storage result; in response to the data storage result being successful, the data copy is asynchronously stored in an object storage resource for hybrid storage of the data to be migrated.

[0110] In the above embodiments, the cloud server receives and responds to data migration requests from the local server, configures distributed storage resources and object storage resources for the data replicas, and performs hybrid storage of the data to be migrated based on the distributed storage resources and object storage resources. This not only enables uninterrupted data migration through interaction with the cloud server, but also allows for cross-data center disaster recovery backup to improve data security, and achieves rapid load balancing of storage resources, reducing data storage costs and meeting diverse storage needs.

[0111] See Figure 7 This application also provides another data migration apparatus 700, which includes:

[0112] The request receiving module 710 is used to receive a data migration request carrying a data directory to be migrated;

[0113] The request-response module 720 is used to respond to a data migration request and determine the data type of the data to be migrated contained in the data directory to be migrated.

[0114] The copy acquisition module 730 is used to acquire a data copy of the data to be migrated based on the data type of the data to be migrated.

[0115] The copy sending module 740 is used to send data copies to the cloud server. The data copies are used by the cloud server to configure distributed storage resources and object storage resources for hybrid storage of the data to be migrated.

[0116] In one embodiment, the copy acquisition module 730 is further configured to, in response to the data type of the data to be migrated being existing data, acquire a copy to be deleted as a data copy based on a first copy set of the data to be migrated directory pre-stored in the local server; and in response to the data type of the data to be migrated being new data, acquire a data copy based on a second copy set of the data to be migrated directory pre-stored in the local server.

[0117] In the above embodiments, the local server receives and responds to a data migration request carrying a directory of data to be migrated, determines the data type of the data to be migrated contained in the directory, and then obtains a copy of the data to be migrated based on the data type. This copy is then sent to the cloud server for the cloud server to configure distributed storage resources and object storage resources, achieving hybrid storage of the data to be migrated. In this way, not only can data migration be achieved without downtime through interaction with the cloud server, but data can also be backed up across data centers to improve data security, meeting diverse storage needs.

[0118] In some embodiments of this application, the data migration apparatus 600 and the data migration apparatus 700 can be implemented as a computer program, which can be implemented in, for example... Figure 8 The computer device shown operates on this device. The computer device's memory can store the various program modules that make up the data migration apparatus 600 and 700, for example, Figure 6 The request receiving module 610, resource configuration module 620, and data migration module 630 are shown; for example, Figure 7 The request receiving module 710, request response module 720, copy acquisition module 730, and copy sending module 740 are shown. The computer program comprised of these various program modules causes the processor to execute the steps in the data migration methods of the various embodiments of this application described in this specification.

[0119] For example, Figure 8 The computer device shown can be used as follows Figure 6 The request receiving module 610 in the data migration device 600 shown executes step S301. The computer device can execute step S302 through the resource configuration module 620. The computer device can execute step S303 through the data migration module 630.

[0120] For example, Figure 8 The computer device shown can be used as follows Figure 7 The request receiving module 710 in the data migration apparatus 700 shown executes step S501. The computer device can execute step S502 through the request response module 720. The computer device can execute step S503 through the copy acquisition module 730. The computer device can execute step S504 through the copy sending module 740.

[0121] The computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external computer devices via a network connection. When the computer program is executed by the processor, it implements a data migration method.

[0122] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0123] In some embodiments of this application, a computer device is provided, including one or more processors; a memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processors using the steps of the data migration method described above. The steps of the data migration method here may be steps from the data migration methods of the various embodiments described above.

[0124] In some embodiments of this application, a computer-readable storage medium is provided, storing a computer program that is loaded by a processor, causing the processor to execute the steps of the data migration method described above. The steps of the data migration method here may be steps from the data migration methods of the various embodiments described above.

[0125] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0126] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0127] The data migration method, apparatus, and computer-readable storage medium provided in the embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data migration method applied in a cloud server, characterized in that, include: Receive a data migration request from the local server, wherein the data migration request carries a copy of the data to be migrated; In response to the data migration request, distributed storage resources and object storage resources are configured for the data replica; The data to be migrated is stored in a hybrid manner based on the distributed storage resources and the object storage resources; the hybrid storage of the data to be migrated based on the distributed storage resources and the object storage resources includes: The data copy is stored in the distributed storage resource to obtain the data storage result; In response to the data storage result being successful, the data copy is asynchronously stored in the object storage resource to perform mixed storage of the data to be migrated.

2. The method as described in claim 1, characterized in that, After configuring distributed storage resources and object storage resources for the data replica in response to the data migration request, the method further includes: In response to the distributed storage resource not meeting the data volume of the data replica, at least one node resource is created in the object storage resource as the first new storage resource of the distributed storage resource; In response to the distributed storage resources containing the first newly added storage resources meeting the data volume, the operation of mixed storage of the data to be migrated is performed.

3. The method as described in claim 2, characterized in that, In response to the distributed storage resource not meeting the data volume requirement of the data replica, at least one node resource is created in the object storage resource as the first new storage resource of the distributed storage resource, including: In response to the distributed storage resource not meeting the data volume of the data replica, at least one node resource is created in the object storage resource; Allocate the at least one node resource to the distributed storage resource as the first newly added storage resource; and The mapping relationship between the node resources and the distributed storage resources is stored, and the mapping relationship is used when the resources are scaled up or down.

4. The method as described in claim 2 or 3, characterized in that, The method further includes: In response to the fact that the distributed storage resources containing the first newly added storage resources do not meet the data volume, target distributed storage resources with excessive load are filtered out. Based on the resource gap required to store the data in the distributed storage resource, determine the target node resources in the target distributed storage resource; Based on the checkpoint information of the target node resource, the target node resource is transferred to the distributed storage resource as a second newly added storage resource of the distributed storage resource; In response to the distributed storage resources, including the first newly added storage resource and the second newly added storage resource, satisfying the data volume, the operation of mixed storage of the data to be migrated is performed.

5. The method as described in claim 4, characterized in that, The step of transferring the target node resource to the distributed storage resource based on the checkpoint information of the target node resource, as a second newly added storage resource of the distributed storage resource, includes: The checkpoint information is generated based on the metadata information pre-stored in the target node resources; The checkpoint information is transmitted to the distributed storage resource through the object storage resource so that the target node resource is added to the distributed storage resource as the second newly added storage resource.

6. A data migration method, applied to a local server, characterized in that, include: Receive data migration requests carrying the data directory to be migrated; In response to the data migration request, determine the data type of the data to be migrated contained in the data directory to be migrated; Based on the data type of the data to be migrated, obtain a data copy of the data to be migrated; The data copy is sent to the cloud server, and the data copy is used by the cloud server to configure distributed storage resources and object storage resources for hybrid storage of the data to be migrated; wherein, the hybrid storage of the data to be migrated includes: The data copy is stored in the distributed storage resource to obtain the data storage result; In response to the data storage result being successful, the data copy is asynchronously stored in the object storage resource to perform mixed storage of the data to be migrated.

7. The method as described in claim 6, characterized in that, The step of obtaining a data copy of the data to be migrated based on the data type of the data to be migrated includes: In response to the fact that the data to be migrated is of existing data, a copy to be deleted is obtained as the data copy based on the first copy set of the data to be migrated directory pre-stored in the local server; In response to the fact that the data to be migrated is new data, the data copy is obtained based on the second copy set of the data to be migrated directory pre-stored in the local server.

8. A data migration device, applied in a cloud server, characterized in that, include: The request receiving module is used to receive data migration requests from the local server, wherein the data migration requests carry a copy of the data to be migrated; The resource configuration module is used to configure distributed storage resources and object storage resources for the data replica in response to the data migration request. A data migration module is configured to perform mixed storage of the data to be migrated based on the distributed storage resources and the object storage resources; the mixed storage of the data to be migrated based on the distributed storage resources and the object storage resources includes: The data copy is stored in the distributed storage resource to obtain the data storage result; In response to the data storage result being successful, the data copy is asynchronously stored in the object storage resource to perform mixed storage of the data to be migrated.

9. A data migration device, applied in a local server, characterized in that, include: The request receiving module is used to receive data migration requests carrying the data directory to be migrated; The request-response module is used to respond to the data migration request and determine the data type of the data to be migrated contained in the data to be migrated directory; The copy acquisition module is used to acquire a data copy of the data to be migrated based on the data type of the data to be migrated; A copy sending module is used to send the data copy to a cloud server. The data copy is used by the cloud server to configure distributed storage resources and object storage resources for hybrid storage of the data to be migrated. The hybrid storage of the data to be migrated includes: The data copy is stored in the distributed storage resource to obtain the data storage result; In response to the data storage result being successful, the data copy is asynchronously stored in the object storage resource to perform mixed storage of the data to be migrated.

10. A computer-readable storage medium, characterized in that, It stores a computer program, which is loaded by a processor to perform the steps of the data migration method according to any one of claims 1 to 7.

11. A computer program product containing instructions, characterized in that, When the instructions are executed on a computer, the computer performs the data migration method according to any one of claims 1 to 7.

Images