Data migration method and system

Abstract

The invention discloses a data migration method and system, and relates to the technical field of cloud computing. The method is applied to a data migration system, the data migration system is used for migrating data stored in a source storage cluster to a target storage cluster, and the data migration system comprises a control node and a plurality of working nodes. The data migration method comprises the steps that a control node obtains the data volume of data to be migrated in a source storage cluster, N first working nodes are determined from multiple working nodes according to the data volume of the data to be migrated, and then a data migration instruction is sent to the N first working nodes. And the N first working nodes migrate the to-be-migrated data from the source storage cluster to the target storage cluster according to the data migration instruction. Wherein the first working node is a working node meeting a preset condition, and N is a positive integer greater than 1. According to the method, the to-be-migrated data is migrated in parallel through the N first working nodes, so that the efficiency of migrating the to-be-migrated data is improved.

Description

[0001] This application claims the priority of the Chinese patent application filed with the State Intellectual Property Office on July 24, 2023, with application number 202310912557.1 and application name “A method, device and other equipment for data processing”, all contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of cloud computing technology, and in particular to a data migration method and system. Background Art

[0003] Hot migration refers to the process of migrating the running applications or required data on the cloud service from one node to another node without interrupting the current cloud service. The node can be a storage node, such as a storage cluster. Usually, a working node in the computer system where the cloud service is deployed migrates the above applications or data from one storage cluster to another. However, when the amount of application or data is large, that is, the working node needs to migrate a lot of data, the migration speed is slow due to the influence of factors such as the bandwidth and load of a single working node, and thus the migration efficiency is low. Summary of the invention

[0004] The present application provides a data migration method and system to solve the problem of slow data migration speed and low migration efficiency during the hot migration of cloud services when the amount of data of the application or data corresponding to the cloud service is large due to factors such as the bandwidth and load of a single working node.

[0005] In a first aspect, the present application provides a data migration method. The data migration method can be applied to a computer system or a computing device applied to the computer system to implement the data migration method, such as, the computer system is a data migration system, and the data migration system is used to migrate data stored in a source storage cluster to a target storage cluster, such as, the computing device is a server, etc. The data migration method includes: a control node obtains the amount of data to be migrated in the source storage cluster, and determines N first working nodes from a plurality of working nodes based on the amount of data to be migrated, and then sends data migration instructions to the N first working nodes. The N first working nodes migrate the data to be migrated from the source storage cluster to the target storage cluster according to the data migration instructions. Among them, the aforementioned first working node is a working node that meets the preset conditions, and N is a positive integer greater than 1.

[0006] In the present application, the data to be migrated is migrated from the source storage cluster to the target storage cluster through N first working nodes, thereby realizing distributed migration of the data to be migrated, and the N first working nodes jointly migrate the data to be migrated, thereby improving the efficiency of migrating the data to be migrated.

[0007] In a possible implementation, the data migration method further includes: the control node obtains a data migration request from a user, where the data migration request includes an identifier of a source storage cluster and / or an identifier of a target storage cluster.

[0008] In a possible example, the control node obtains a trigger operation of a user on a control component on a user display interface, and obtains a data migration request of the user in response to the trigger operation.

[0009] For example, the identifier may be the name of the storage cluster or an identity document (ID).

[0010] In the present application, the control node receives the user's clear identification of the source storage cluster and / or the identification of the target storage cluster, and then the control node can instruct the working node to migrate the data in the source storage cluster to the target storage cluster based on the identification to achieve the migration of the data in the source storage cluster.

[0011] In a possible scenario, the data migration request further includes an identifier of the data to be migrated, which may be a name of the data, an ID of a sector, a storage block or a cloud hard disk where the data to be migrated is located, or the like.

[0012] In a possible example, the storage performance of the target storage cluster is better than the storage performance of the source storage cluster. The storage performance may include storage capacity, bandwidth, or latency.

[0013] In one possible implementation, the control node determines N first working nodes from multiple working nodes, including: the control node calculates the amount of migratable data of the multiple working nodes per unit time, and sorts the multiple working nodes according to the amount of migratable data of the multiple working nodes, and then the control node selects the first N working nodes as the first working nodes according to the sorting results.

[0014] Exemplarily, the above sorting result is obtained by arranging the control nodes in descending order according to the amount of data that can be migrated by the working nodes per unit time.

[0015] In the present application, N first working nodes are used to jointly migrate the data to be migrated, thereby improving the efficiency of migrating the data to be migrated. Furthermore, since the first working nodes are the N working nodes ranked first in terms of the amount of data that can be migrated per unit time, the efficiency of migrating the data to be migrated can be further improved by using the N first working nodes to migrate the data to be migrated.

[0016] In a possible scenario, the amount of data that can be migrated within the unit time may be the maximum amount of data that can be migrated within the unit time.

[0017] In a possible example, the control node calculates the amount of data that can be migrated by multiple working nodes per unit time, including: the control node calculates the amount of data that can be migrated by multiple working nodes per unit time according to the bandwidth, historical migration information and migration task information of the multiple working nodes, wherein the historical migration information is used to indicate the failure or success information of the multiple working nodes in executing historical migration tasks, and the migration task information is used to indicate the migration start time of the data to be migrated.

[0018] For example, the control node may use a weighted sum method or a machine learning model to calculate the migrateable data of the working node per unit time.

[0019] In the present application, the control node calculates the amount of data that can be migrated by the working node per unit time from the historical dimension (historical migration information and network bandwidth) and the current dimension (migration task information), which is conducive to more accurately determining the amount of data that can be migrated by the working node per unit time. Then the control node selects the N first working nodes ranked at the top in terms of the amount of data that can be migrated per unit time obtained by the above calculation method for data migration, which can improve the efficiency and success rate of migrating the data to be migrated.

[0020] In a possible implementation, the control node obtains the amount of data to be migrated in the source storage cluster, including: the control node obtains a bitmap of the source storage cluster, and then based on the bitmap, uses valid data in the source storage cluster as data to be migrated, and the bitmap is used to indicate whether the data stored in the source storage cluster is valid data.

[0021] In the present application, only valid data in the source storage cluster indicated by the user, i.e., data to be migrated, is migrated, which reduces the amount of data to be migrated, thereby speeding up the migration speed and improving the efficiency of data migration. It also avoids the problem of migrating invalid data to the target storage cluster and wasting the storage resources of the target storage cluster, thereby improving the utilization rate of the storage space in the target storage cluster.

[0022] In a possible implementation, during data migration, if a user accesses target data in the data to be migrated, the user's input / output load is mounted on the target storage cluster, and the above data migration method further includes: the second working node obtains a snapshot bitmap of the target storage cluster, and determines whether the target data is stored in the target storage cluster according to the snapshot bitmap. If the target data is stored in the target storage cluster, the second working node reads the target data from the target storage cluster; if the target data is stored in the source storage cluster, the second working node reads the target data from the source storage cluster.

[0023] In this application, if during data migration, the user needs to access part of the data to be migrated (target data), the control node can determine whether the target data is stored in the source storage cluster or the target storage cluster by querying the snapshot bitmap, and then access the target data in the corresponding storage space. This avoids the problem of data disorder caused by the inconsistency between the actually accessed data and the target data due to unclear storage location of the target data, improves the accuracy of accessing the target data during data migration, and does not affect the normal operation of the business, that is, hot migration is achieved.

[0024] In a possible scenario, if the target data is stored in the source storage cluster, the second working node reads the target data from the source storage cluster and then migrates the target data to the target storage cluster.

[0025] In the present application, the second working node migrates the target data stored in the source storage cluster to the target storage cluster according to the user's access requirements (the user accesses the target data in the data to be migrated), thereby realizing migration according to the user's requirements.

[0026] In the second aspect, the present application provides a data migration system. The data migration system is used to migrate data stored in a source storage cluster to a target storage cluster, and the data migration system includes a control node and multiple working nodes, such as the data migration system is a computer cluster. The data migration system includes: a control node, which is used to obtain the data volume of the data to be migrated in the source storage cluster, and determine N first working nodes from multiple working nodes based on the data volume of the data to be migrated. Among them, the first working node is a working node that meets the preset conditions, and N is a positive integer greater than 1. Then, the control node sends a data migration instruction to the N first working nodes. The N first working nodes are used to migrate the data to be migrated from the source storage cluster to the target storage cluster according to the data migration instruction.

[0027] In a possible implementation manner, the control node is specifically configured to: obtain a data migration request from a user, where the data migration request includes an identifier of a source storage cluster and / or an identifier of a target storage cluster.

[0028] In one possible implementation, the control node is specifically used to: calculate the amount of migratable data of multiple working nodes per unit time, sort the multiple working nodes according to the amount of migratable data of the multiple working nodes, and then select the first N working nodes as the first working nodes according to the sorting results.

[0029] In a possible implementation, the control node is further specifically used to calculate the amount of data that can be migrated by multiple working nodes per unit time according to the bandwidth, historical migration information and migration task information of the multiple working nodes, wherein the historical migration information is used to indicate the failure or success information of the multiple working nodes in executing historical migration tasks, and the migration task information is used to indicate the migration start time of the data to be migrated.

[0030] In a possible implementation, the control node is specifically used to: obtain a bitmap of the source storage cluster, and according to the bitmap, the control node uses valid data in the source storage cluster as data to be migrated; the bitmap is used to indicate whether the data stored in the source storage cluster is valid data.

[0031] In a possible implementation, during data migration, if a user accesses target data in the data to be migrated, the user's input / output load is mounted on the target storage cluster, and the second working node among the multiple working nodes is specifically used to: obtain a snapshot bitmap of the target storage cluster, and determine whether the target data is stored in the target storage cluster according to the snapshot bitmap. The second working node is used to read the target data from the target storage cluster if the target data is stored in the target storage cluster. The second working node is used to read the target data from the source storage cluster if the target data is stored in the source storage cluster, and migrate the target data to the target storage cluster.

[0032] In a third aspect, the present application provides a chip, which includes: a processor and a power supply circuit; the power supply circuit is used to supply power to the processor, and the processor is used to execute the method in the first aspect or any possible implementation of the first aspect.

[0033] In a fourth aspect, the present application provides a computing device cluster. The computing device cluster includes at least one computing device, the computing device includes a memory and a processor, the memory is used to store computer instructions; when the processor executes the computer instructions, the method in the first aspect or any possible implementation of the first aspect is implemented.

[0034] In a fifth aspect, the present application provides a computer-readable storage medium. The storage medium stores a computer program or instruction, and when the computer program or instruction is executed by a processing device, the method in the first aspect or any possible implementation of the first aspect is implemented.

[0035] In a sixth aspect, the present application provides a computer program product. The computer program product includes a computer program or instructions, and when the computer program or instructions are executed by a processing device, the method in the first aspect or any possible implementation of the first aspect is implemented.

[0036] The beneficial effects of the second to sixth aspects above can be referred to the description of the first aspect or any implementation method of the first aspect, and will not be elaborated here.

[0037] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of the process of hot migration of a cloud hard disk is provided;

[0039] Figure 2 An application scenario diagram of a computer system provided for this application;

[0040] Figure 3 A flowchart of the data migration method provided for this application;

[0041] Figure 4 A schematic diagram of a flow chart of an indication migration method provided for this application;

[0042] Figure 5 A flowchart of a data access method provided for this application;

[0043] Figure 6 A structural diagram of a data migration system provided for this application;

[0044] Figure 7 A schematic diagram of the structure of a computing device provided for this application;

[0045] Figure 8 A schematic diagram of the structure of a computer cluster provided for this application;

[0046] Fig. 9 A schematic diagram of the connection between computing devices provided for this application. DETAILED DESCRIPTION

[0047] To facilitate understanding, the technical terms involved in this application are first introduced.

[0048] Cloud computing platform refers to a platform that provides computing resources and services through the Internet. Based on virtualization technology, cloud computing platform abstracts and aggregates computing, storage, network and other resources, allowing users to use these resources on demand without having to worry about the underlying physical hardware and infrastructure.

[0049] In cloud computing, a region refers to a geographical area divided by a cloud service provider around the world. Each region is usually composed of one or more data centers that can provide a variety of cloud services, such as computing, storage, and databases.

[0050] Cloud hard disk is a service that provides persistent block storage for computing servers such as elastic cloud servers (ECS) and bare metal servers (BMS). Cloud hard disk can use multiple technologies such as data redundancy and caching to provide high availability and persistence, as well as stable low-latency services.

[0051] As more and more businesses are deployed on the cloud computing platform, the corresponding storage data is increasing. In order to avoid storing too much data in storage cluster a under the cloud computing platform, resulting in storage performance degradation or data explosion of storage cluster a, it is necessary to migrate the storage data, such as migrating the storage data from storage cluster a to storage cluster b.

[0052] like Figure 1 As shown, Figure 1 A schematic diagram of the hot migration process of a cloud hard disk is provided. Figure 1 The figure shows a cloud hard disk hot migration method, that is, migrating the data in the cloud hard disk (virtual machine disk) from the storage cluster 120 to the storage cluster 130 for storage. During the migration process, the business is not interrupted and the user is not aware of it. Figure 1 The central control node 111 and the working node 112 belong to the same computing cluster 110 . The computing cluster 110 and the storage cluster 120 and the storage cluster 130 are located in the same data center or in different data centers.

[0053] Figure 1 The central control node 111 determines a working node 112 to perform data migration, and the working node 112 migrates the data of the cloud hard disk 121 in the storage cluster 120 to the cloud hard disk 131 in the storage cluster 130 .

[0054] The following is a specific example of a cloud disk migration process, which includes the following steps ①-⑤.

[0055] ①: Read the source disk file (data in the cloud hard disk 121) in the storage cluster 120. The source disk file includes the format, size, content, etc. of the file in the disk.

[0056] ②: Based on the source disk file, create an empty image file (target disk file, that is, cloud hard disk 131 ) in storage cluster 130 that is consistent with cloud hard disk 121 in storage cluster 120 .

[0057] ③: Set the target disk file and the source disk file as mirrors, so that during the migration process, the user's modification of the source disk file in the storage cluster 120 is also synchronized to the target disk file in the storage cluster 130 to ensure data consistency.

[0058] ④: Migrate the baseline data on the source disk file to the target disk file.

[0059] ⑤: After the baseline data migration is completed, the IO request of the upper-level virtual machine (or container, etc.) is temporarily suspended, and the mapping relationship between the virtual machine (or container, etc.) and the target disk file is modified to complete the entire migration process.

[0060] The control node 111 uses a working node to migrate the source disk files in the storage cluster 120. When the source disk files are larger, the migration speed is slower due to restrictions such as migration speed and migration strategy, that is, the migration efficiency is lower.

[0061] In summary, when the amount of data of the application or data corresponding to the cloud service is large, the above solution will lead to slow data migration speed and low migration efficiency due to factors such as the bandwidth and load of a single working node.

[0062] Based on this, the present application provides a data migration method, which is applied to a data migration system, and the data migration system is used to migrate data stored in a source storage cluster to a target storage cluster. The data migration system includes a control node and multiple working nodes. The data migration method includes: the control node obtains the data volume of the data to be migrated in the source storage cluster, and determines N first working nodes from multiple working nodes according to the data volume of the data to be migrated, and then sends data migration instructions to the N first working nodes. The N first working nodes migrate the data to be migrated from the source storage cluster to the target storage cluster according to the data migration instructions. Among them, the aforementioned first working node is a working node that meets the preset conditions, and N is a positive integer greater than 1.

[0063] In the present application, the data to be migrated is migrated from the source storage cluster to the target storage cluster through N first working nodes, thereby realizing distributed migration of the data to be migrated, and the N first working nodes jointly migrate the data to be migrated, thereby improving the efficiency of migrating the data to be migrated.

[0064] The data migration method provided in this application can be applied to Figure 2 The data migration system shown in Figure 1 is as follows. Figure 2 As shown, Figure 2 This is an application scenario diagram of a data migration system provided in the present application. The data migration system includes a control node 210 and a working node cluster 220. The control node 210 and the working node cluster 220 can communicate with each other in a wired manner or in a wireless manner.

[0065] In the present application, the control node 210 is used to schedule the working nodes in the working node cluster, and the working node cluster is used to migrate data according to the instruction of the control node 210.

[0066] The working node cluster 220 may include: a plurality of working nodes. For example, the plurality of working nodes include working node 221, working node 222, and working node 223. The working node may also be referred to as a computing device or a working device, and the computing device may be used to perform data migration. The control node may also be referred to as a computing device or a control device, and the computing device may be used to perform scheduling of the working node.

[0067] The wired communication may be: Ethernet, optical fiber, and various peripheral component interconnect express (PCIe) buses provided inside the data migration system for connecting the control node 210 and the working node cluster 220.

[0068] The wireless communication may be: Internet, wireless fidelity (WIFI) and ultrawide band (UWB) technology.

[0069] In a possible scenario, the data migration system is used to migrate data stored in a source storage cluster to a target storage cluster.

[0070] The source storage cluster and the target storage cluster may include a variety of storage devices, such as hard disk drives (HDD), solid state drives (SSD), optical disks, memory cards, tapes, etc. The storage cluster is used to store data generated or required by cloud services.

[0071] In a possible example, the source storage cluster is mounted on a virtual machine corresponding to the cloud service, and the input / output (I / O) load of the cloud service is on the source storage cluster.

[0072] In a possible embodiment, the data migration system includes a source storage cluster 230 and a target storage cluster 240 .

[0073] In another possible embodiment, the source storage cluster 230 and the target storage cluster 240 are independently configured and only communicate with the data migration system to transfer data.

[0074] The above embodiments are merely examples and should not be construed as limiting the present application.

[0075] In a possible example, the data migration system may further include a terminal 250 , a terminal 260 , and a terminal 270 .

[0076] Terminal 250, terminal 260, and terminal 270 can be a mobile phone, a tablet computer, a handheld computer, a personal computer (PC), a cellular phone, a personal digital assistant (PDA), a wearable device (such as a smart watch), a smart home device (such as a television), a car computer, a game console, and an augmented reality (AR) / virtual reality (VR) device, etc. This application does not impose any special restrictions on the specific device form of terminal 250, terminal 260, and terminal 270.

[0077] In another possible example, the terminal 250 , the terminal 260 , and the terminal 270 are independently configured and are communicatively connected to the data migration system.

[0078] It should be noted that Figure 2 The illustrated data migration system architecture is only an example, and the device type or number within the system can be configured according to actual needs, and the embodiments of the present application are not limited thereto. For example, the data migration system may also include more control nodes or working nodes.

[0079] The implementation of the data migration method provided in the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0080] Here, the data migration method of the embodiment of the present application is as follows: Figure 2 The control node 210 and the working node cluster 220 shown are used as an example for explanation. Figure 3 A flow chart of a data migration method provided for the present application. Among them, the control node 310 is used to implement the function of the control node 210, and the working node 320, the working node 330 and the working node 340 are used to implement the function of the working node cluster 220. In this embodiment, the control node 310 can also be referred to as a control device or a computing device, and the working node 320, the working node 330 or the working node 340 can also be referred to as a working device or a computing device. The data migration system can be connected to the source storage cluster and the target storage cluster via a bus, and the bus can refer to a universal serial bus (USB) or a unified bus (Ubus or UB). In some examples, the bus can also be referred to as a computer express link (CXL), a cache coherent interconnect for accelerators (CCIX) or other names. In this embodiment, the source storage cluster 230 and the target storage cluster 240 are independently set.

[0081] In the first possible application scenario, the control node 310 schedules the working node 320, the working node 330 or the working node 340, so that the working node 320, the working node 330 or the working node 340 migrates the valid data stored in the source storage cluster to the data in the target storage cluster, thereby reducing the capacity of the stored data. For example, data is migrated from cloud hard disk a in the source storage cluster to cloud hard disk b in the target storage cluster.

[0082] In a second possible application scenario, control node 310 schedules working node 320, working node 330 or working node 340, so that working node 320, working node 330 or working node 340 migrates all or part of the data stored in the source storage cluster to the data in the target storage cluster, thereby adjusting the load of the source storage cluster.

[0083] In a third possible application scenario, the control node 310 replaces the storage cluster according to the business needs or user needs in the cloud service, and then schedules the working node 320, the working node 330 or the working node 340, so that the working node 320, the working node 330 or the working node 340 migrates the data in the source storage cluster to the target storage cluster. The storage performance of the target storage cluster is better than the storage performance of the source storage cluster. For example, the target storage cluster includes the disk type required by the user (such as SSD) or the user purchases a cloud hard disk (disk) on the target storage cluster.

[0084] The above three possible application scenarios are only examples provided in this embodiment and should not be construed as limitations on this application. In other possible examples, the control node 310 schedules the working node 320, the working node 330 or the working node 340 to migrate data in the source storage cluster to the target storage cluster according to the target storage cluster newly accessed by the user.

[0085] In a possible embodiment, the control node 310 obtains a data migration request from a user. The data migration request includes an identifier of a source storage cluster and / or an identifier of a target storage cluster.

[0086] The above identifier may be the name or ID of the storage cluster, etc.

[0087] In a possible implementation, the control node 310 receives a trigger operation of a user on a control component on a user display interface, and obtains a data migration request of the user in response to the trigger operation.

[0088] Exemplarily, the triggering operation may be an identifier of a source storage cluster and / or an identifier of a target storage cluster input or selected by a user through a control component in a user display interface on terminal 250 , terminal 260 , or terminal 270 .

[0089] In a possible scenario, the data migration request further includes an identifier of the data to be migrated. The identifier of the data to be migrated may be the name of the data, the sector where the data to be migrated is located, the ID of the storage block or the cloud hard disk, etc.

[0090] See also Figure 3 The data migration method provided in this embodiment includes the following steps S310 to S340.

[0091] Step S310 : The control node 310 obtains the amount of data to be migrated in the source storage cluster 230 .

[0092] With respect to the control node 310 obtaining the amount of data to be migrated in the source storage cluster 230 , three possible implementations are provided below.

[0093] In a first possible implementation manner, the control node 310 receives the data name of the data to be migrated indicated by the user, queries metadata according to the data name of the data to be migrated, and further obtains the data volume of the data to be migrated.

[0094] The metadata is stored in a specified area in the source storage cluster 230, and the metadata records information about the data in the source storage cluster 230, such as the description, format, size, etc. of the data. The control device 221 accesses the specified area to determine the metadata information corresponding to the data name of the data to be migrated, thereby obtaining the data volume of the data to be migrated.

[0095] In a second possible implementation, the source storage cluster 230 is managed by a database system, the control node 310 obtains the identifier of the data to be migrated indicated by the user, and uses a database management tool to query the size of the data to be migrated, that is, the data volume.

[0096] The identification of the data to be migrated includes the name of the data to be migrated or the identity document (ID) of the sector / disk block, etc., and the database management tool is such as MySQL Workbench and Oracle Enterprise Manager. The control node 310 uses the data name of the data to be migrated in MySQL Workbench and Oracle Enterprise Manager to query the data volume of the data to be migrated.

[0097] In a third possible implementation manner, the control node 310 obtains the sector / disk block ID of the data to be migrated indicated by the user, and directly reads the data of the corresponding sector or disk block to determine the data amount of the data to be migrated.

[0098] Step S320: The control node 310 determines N first working nodes from the multiple working nodes according to the amount of data to be migrated.

[0099] Wherein, N is a positive integer, and the first working node is: a working node that meets preset conditions among multiple working nodes.

[0100] In a possible example, the sum of the amounts of data that can be migrated of the N first working nodes is greater than or equal to the amount of data to be migrated.

[0101] In a possible scenario, the working node that meets the preset condition may be a working node whose amount of data that can be migrated per unit time meets the condition.

[0102] In the case where the preset conditional working node is a working node whose amount of data that can be migrated per unit time meets the condition, a possible embodiment is provided below for the control node 310 to determine the contents of N first working nodes from multiple working nodes.

[0103] The control node 310 calculates the amount of data that can be migrated by multiple working nodes per unit time, and then sorts the multiple working nodes according to the amount of data that can be migrated by the multiple working nodes per unit time to obtain a sorting result, and then selects the first N working nodes as the first working nodes according to the sorting result.

[0104] In some optional examples, the amount of data that can be migrated by a working node per unit time may also be referred to as the data migration capability of the working node. It should be understood that the data migration capability may include one or more indicators: the number of input / output operations per second (IOPS), bandwidth, the amount of data per atomic operation, etc.

[0105] Exemplarily, the amount of data that can be migrated by a working node within a unit time is the maximum amount of data that can be migrated within a unit time.

[0106] In a possible scenario, the above sorting may be in the order of the amount of data that can be migrated per unit time from large to small, that is, the control node 310 arranges the working nodes in descending order according to the amount of data that can be migrated per unit time.

[0107] In a possible example, N may be a value such as 1 / 2 / 3 / 4 / 10 / 16 / 32, which is not limited in this application.

[0108] The following is explained using N=2 as an example: control node 310 determines working node 330 and working node 340 as the first working node from working node 320, working node 330 and working node 340 based on the amount of data that can be migrated per unit time by working node 320, working node 330 and working node 340.

[0109] The above contents are merely examples provided in this embodiment and should not be construed as limiting the present application. In other possible examples, N may also be other values, such as 64, etc., which will not be described in detail here.

[0110] like Figure 3 As shown, the control node 310 manages multiple working nodes. In this embodiment, the working nodes are used to migrate the data to be migrated in the source storage cluster 230 to the target storage cluster 240. Due to the differences in hardware of each working node (IOPS, bandwidth, and data volume of each atomic operation), different tasks being run (different resource occupation), and different historical migrated data information (historical IOPS), each working node has different migration capabilities for the migrated data. Therefore, in order to migrate data efficiently, the control node 310 sorts the multiple working nodes according to the migration capabilities of the working nodes for the migrated data to obtain a sorting result.

[0111] In the present application, the amount of data that can be migrated by the first working node per unit time is the first N of the amounts of data that can be migrated by multiple working nodes per unit time. Therefore, by using the N first working nodes to migrate the data to be migrated to the target storage cluster 240, the speed of migrating the data to be migrated from the source storage cluster 230 to the target storage cluster 240 can be improved, thereby improving the efficiency of data migration.

[0112] In one possible example, the control node 310 calculates the amount of data that can be migrated per unit time for multiple working nodes, including: the control node 310 uses one or more of the network information of the working nodes, the historical migration information of the working nodes, and the migration task information to determine the amount of data that can be migrated per unit time for each of the multiple working nodes.

[0113] The network information of the working node includes one or more combinations of bandwidth, network delay, network topology, and IOPS.

[0114] The historical migration information of the working node is used to indicate: the failure or success of the working node for the historical migration task (whether the IOPS is 0, whether the amount of data for each atomic operation is 0), the duration of the historical migration task, or the start time of the historical migration task, or a combination of one or more of the following.

[0115] The migration task information is used to indicate: the start time of the migration task (a task of migrating the data to be migrated), the type or amount of the data to be migrated, or a combination of one or more items.

[0116] The start time of the above historical migration task and the start time of the migration task represent the time when the task is constructed in the control node 310 or the time when it is added to the migration queue. The control node 310 distributes the task to the working node in sequence according to the above migration queue to instruct the working node to migrate the data corresponding to the task. The time consumption of the historical migration task represents the time from the time when the control node 310 constructs the task to the time when the working node completes the migration of the data indicated by the historical migration task.

[0117] In the present application, the control node calculates the amount of data that can be migrated by the working node per unit time from the historical dimension (historical migration information and network bandwidth) and the current dimension (migration task information), which is conducive to more accurately determining the amount of data that can be migrated by the working node per unit time. Then the control node selects the N first working nodes ranked at the top in terms of the amount of data that can be migrated per unit time obtained by the above calculation method for data migration, which can improve the efficiency and success rate of migrating the data to be migrated.

[0118] In other embodiments of the present application, the control node 310 may also determine the amount of data that can be migrated by the working node per unit time from indicators such as hardware information of the working node. The hardware information may indicate the IOPS of the working node, the amount of data per atomic operation, bandwidth, etc.

[0119] In a possible example, the control node 310 adopts a set processing strategy to process one or more of the network information of the working node, the historical migration information of the working node, and the migration task information to obtain the above-mentioned sorting result.

[0120] For example, the control node 310 uses a set processing strategy to process one or more of the network information of the working node, the historical migration information of the working node, and the migration task information to obtain the score of the working node (that is, the amount of data that can be migrated per unit time). Then, the control node 310 sorts the working nodes according to the scores to obtain a sorting result.

[0121] Regarding the content of the processing strategy, two possible embodiments are shown below, which are not described in detail here. And regarding the content of the control node 310 determining the top N first working nodes in the sorting result, please refer to the two possible embodiments provided below, which are not described in detail here.

[0122] Step S330: The control node 310 sends a data migration instruction to the N first working nodes.

[0123] Exemplarily, the control node 310 sends a data migration instruction to each first working node, and the data migration instruction received by each first working node indicates that there is a difference in the migrated data.

[0124] In a possible scenario, the data migration received by each first working node carries a different logical address (logical block address), thereby instructing the first working node to migrate the data indicated by the logical address.

[0125] The above-mentioned logical address includes: a base address and an address offset, and the address offset is used to indicate: the relative storage position of the data to be migrated by each first working node in the first storage space compared to the base address.

[0126] In a possible example, the base address received by each first computing device is the same, but the received address offsets are different.

[0127] For the specific content of the control node 310 sending the data migration instruction to the N first working nodes, please refer to the following embodiment of the control node 310 sending the data migration instruction to the N first working nodes, which is not described in detail here.

[0128] Step S340: N first working nodes migrate the data to be migrated from the source storage cluster to the target storage cluster according to the data migration instruction.

[0129] Each of the N first working nodes migrates part of the data to be migrated stored in the logical address to the target storage cluster according to the logical address indicated in the data migration instruction.

[0130] The following shows an embodiment in which the first working node migrates the data to be migrated from the source storage cluster to the target storage cluster according to the data migration instruction. In this embodiment, the execution of the working node 330 is used as an example for explanation, and the data migration request indicates the identifiers of the source storage cluster and the target storage cluster, such as ID. This embodiment may include the following steps ① to ⑤.

[0131] Step ①: The working node 330 determines the cloud hard disk a that needs to be migrated, and the cloud hard disk a is mounted on the virtual machine a.

[0132] Step ②: The working node 330 creates a cloud disk b for the virtual machine a. The cloud disk b includes a system disk or a data disk. The system disk usually contains an operating system and related system files, while the data disk is used to store application data, etc. The system disk or data disk data belongs to the target storage cluster 240.

[0133] Exemplarily, the cloud hard disk b may be purchased from a cloud service provider, or created accordingly on the target storage cluster 240 .

[0134] Step 3: The working node 330 obtains the image data of virtual machine a from the object storage service (OBS), and writes the image data of virtual machine a to the above-mentioned cloud hard disk b. The image data includes operating system image data and application image data, etc. For example, the operating system image data includes the installation files and related configurations of the operating system. The operating system image data and application image data are stored in the source storage cluster 230.

[0135] In one possible scenario, after writing the image data of virtual machine a to cloud hard disk b, working node 320 mounts cloud hard disk b to virtual machine a, migrates the I / O load of the virtual machine from cloud hard disk a to cloud hard disk b, and cloud hard disk a unmounts virtual machine a.

[0136] Step ④: The working node 330 uses the virtual block storage service (VBS) to enable lazy loading, that is, when the virtual machine a is started, the necessary data of the operating system is loaded into the memory instead of loading all at once, which is conducive to speeding up the startup speed of the virtual machine and reducing resource usage.

[0137] During lazy loading, the system first loads the basic data required by the operating system and then starts the virtual machine to speed up the startup time.

[0138] Step 5: After virtual machine a is started, if the customer needs to access data that is not loaded into the memory, such as large application files or data files, this will trigger the on-demand loading mechanism. The system will gradually load the data into the memory according to the customer's access needs to ensure that the customer can obtain the required data in a timely manner.

[0139] In a possible example, the operating system image data and the application image data are data indicated by a logical address in a data migration instruction sent by the control node 310 to the working node 330 .

[0140] In one possible scenario, during data migration, the I / O loads corresponding to the business are all mounted on the source storage cluster, that is, during data migration, when the business accesses data, it first accesses the source storage cluster. If the source storage cluster does not have the corresponding data, it accesses the target storage cluster.

[0141] Exemplarily, before a preset ratio of data in the source storage cluster is migrated to the target storage cluster, the I / O load corresponding to the service is mounted on the source storage cluster. The preset ratio is, for example, 90% or 100%.

[0142] With respect to the control node 310 sending a data migration instruction to N first working nodes, a possible embodiment is provided below.

[0143] like Figure 4 As shown, Figure 4 A schematic diagram of an indication migration method provided in the present application. After the control node 310 determines N first working nodes, the control node 310 splits the data to be migrated into multiple sub-data, and one first working node migrates one sub-data. Then, the control node 310 sends the logical address of the storage space where each sub-data is located to the corresponding first working node, thereby instructing the first working node to migrate the data corresponding to the logical address to the target storage cluster 240, so that the N first working nodes in the first order can jointly migrate the data to be migrated to the target storage cluster 240.

[0144] For example, the control node 310 determines that the working node 330 migrates the sub-data a, and the working node 340 migrates the sub-data b. The control node 310 sends the base address "1000" and the address offsets "0-3" and "4-7" corresponding to the sub-data a to the working node 330. The working node 330 determines that the address of the sub-data a indicated by the logical address is "1000:0-1000:3" and "1000:4-1000:7" based on the base address "1000" and the address offsets "0-3" and "4-7". In addition, the control node 310 sends the base address "1000" and the address offset "8-12" corresponding to the sub-data b to the working node 340. The working node 340 determines that the address of the sub-data b indicated by the logical address is "1000:8-1000:12" based on the base address "1000" and the address offset "8-12".

[0145] The working node 330 performs address conversion on the logical addresses "1000:0-1000:3" and "1000:4-1000:7" through the memory management unit (MMU) to obtain the corresponding physical addresses, and then migrates the sub-data a indicated by the physical addresses to the target storage cluster 240. In addition, the working node 340 performs address conversion on the logical addresses "1000:8-1000:12" through the MMU to obtain the corresponding physical addresses, and then migrates the sub-data a indicated by the physical addresses to the target storage cluster 240.

[0146] In the present application, after converting the received logical address, the first working node can directly access part of the data to be migrated, and then migrate part of the data from the source storage cluster 230 to the target storage cluster 240, avoiding the first working node from determining the address of the aforementioned part of the data, thereby improving the efficiency of data migration.

[0147] In a possible embodiment, the storage performance of the target storage cluster 240 is better than the storage performance of the source storage cluster 230. The storage performance includes: one or more combinations of storage capacity, bandwidth, or latency.

[0148] Example 1: the storage capacity of the target storage cluster 240 is 100T, the storage capacity of the source storage cluster 230 is 50T, and the storage capacity of the target storage cluster 240 is greater than the storage capacity of the source storage cluster 230 .

[0149] Example 2: The target storage cluster 240 is an SSD cluster, and the source storage cluster 230 is an HDD cluster. Since the bandwidth and latency of SSD are better than those of HDD, the bandwidth and latency of the target storage cluster 240 are better than those of the source storage cluster 230 .

[0150] Two possible embodiments are provided below for the above processing strategy.

[0151] In a first possible embodiment, the above-set processing strategy may be a weighted sum strategy.

[0152] The control node 310 uses a weighted summation strategy to determine the values ​​of the network information of the working node, the historical migration information of the working node, and the migration task information, and then based on the weights of each indicator (the network information of the working node, the historical migration information of the working node, and the migration task information), weighted sums the values ​​to obtain the score of each working node.

[0153] Taking the value of the network information of the working node determined by the control node 310 as an example, the value of the network information of the working node, the historical migration information of the working node, and the migration task information determined by the control node 310 is described. The control node 310 normalizes the network information of the working node to obtain the value of the network information of the working node.

[0154] For example, if the network information of the working node includes bandwidth, the control node 310 may normalize the bandwidth according to the following Table 1 to obtain a value corresponding to the network information. The bandwidth is 5 Gbps, and the control node 310 may assign a value of 0.7 to the network information indicator according to the following Table 1.

[0155] Table 1

[0156] Bandwidth (unit: Gbps) Assignment 0-1 0.2 1-3 0.5 4-10 0.7 10-∞ 1

[0157] It is worth noting that the above Table 1 is only an example and should not be understood as a limitation of the present application. The above range is inclusive to the right, that is, if the bandwidth is 1 Gbps, it is assigned a value of 0.2. In a possible example, the above range can be more fine-grained. For example, if the bandwidth is 0-1 Gbps, it is assigned a value of 0.2, and if the bandwidth is 1-2 Gbps, it is assigned a value of 0.3, etc., which will not be repeated here.

[0158] The control node 310 normalizes the historical migration information and migration task information of the working nodes by normalizing the network information as described above, thereby obtaining corresponding values.

[0159] The control node 310 performs weighted summation on the values ​​of the network information of the working node, the historical migration information of the working node, and the migration task information to obtain the score of the working node. The weights of the network information of the working node, the historical migration information of the working node, and the migration task information are set by the user according to actual needs.

[0160] For example, the weight corresponding to the network information is 0.4, the weight corresponding to the historical migration information is 0.4, and the weight corresponding to the migration task information is 0.2. If the normalized value corresponding to the network information is 0.7, the normalized value corresponding to the historical migration information is 0.6, and the normalized value corresponding to the migration task information is 0.5, the control node 310 performs a weighted sum (0.4x0.7+0.4x0.6+0.2x0.5) and obtains a score of 0.62.

[0161] The control node 310 determines the scores of the multiple working nodes. The content of determining the score of a working node can be referred to above, and will not be repeated here.

[0162] In a second possible embodiment, the processing strategy set above may be a machine learning model.

[0163] The above-mentioned machine learning models may include decision trees, random forests, extreme gradient boosting (XGBoost), simulated annealing, recurrent neural networks (RNN), Transformers, long short-term memory networks (LSTM), etc.

[0164] Take the Transformer in the machine learning model as an example. The control node 310 inputs the network information of the working node, the historical migration information of the working node, and the migration task information into the Transformer, and vectorizes the network information of the working node, the historical migration information of the working node, and the migration task information through the embedding layer in the Transformer to obtain the corresponding vector. This vector enables the Transformer to better understand and process the above text data. Through the encoder (encoder) and decoder (decoder) in the Transformer, the score of each working node is obtained.

[0165] Among them, the encoder converts the input vector into a high-dimensional context representation, and the decoder generates an output sequence based on this context representation, such as the score of the working node.

[0166] Regarding the control node 310 determining the contents of the N first working nodes ranked first in the sorting result, two possible embodiments are provided below.

[0167] In a possible embodiment, the control node 310 determines the N first working nodes ranked first in the sorting result, including: the control node 310 determines the N first working nodes ranked first according to the amount of data that can be migrated by the working nodes in a unit time. The sum of the maximum amount of data that can be migrated of the N first working nodes ranked first satisfies the amount of data to be migrated.

[0168] It is worth noting that the amount of data that can be migrated by a working node is configured by the user according to actual needs. For example, the amount of data that can be migrated by a working node is 500M or 1G.

[0169] Exemplarily, the control node 310 determines, in order of the sorting results, whether the amount of data that can be migrated from the first N working nodes satisfies the amount of data to be migrated. If it is determined that the amount of data that can be migrated from the N working nodes satisfies the amount of data to be migrated, the N working nodes are determined to be the first working nodes.

[0170] For example, if the amount of data to be migrated is 10G and the amount of data that can be migrated by the working node is 3G, the control node 310 judges the order of the sorting results in turn, and judges that the sum of the amount of data that can be migrated by the top four working nodes in the sorting results meets the amount of data to be migrated, then the working nodes ranked in the top 4 in the sorting results are determined to be the first working nodes.

[0171] In the above example, the amount of data that can be migrated by each working node is the same. In other possible embodiments, the amount of data that can be migrated by each working node may be different. In the case where the amount of data that can be migrated by each working node is different, the control node 310 can also determine the first working node according to the content of the above implementation method, which is not repeated here.

[0172] In another possible embodiment, the control node 310 determines the N first working nodes ranked first in the sorting result, including: the control node 310 divides the data to be migrated into multiple copies, and determines the N first working nodes ranked first in the sorting result according to the number of copies of the data to be migrated. The amount of sub-data after the data to be migrated is divided into multiple copies is less than or equal to the amount of data that can be migrated by the first working node.

[0173] Exemplarily, the control node 310 divides the data to be migrated into a parts, thereby obtaining the data volume of each sub-data. The control node 310 determines whether the amount of data that can be migrated of each working node in the first a working nodes in the sorting result is greater than or equal to the data volume of each sub-data. If the amount of data that can be migrated of each working node is greater than the data volume of each sub-data, the working nodes ranked in the first a in the sorting result are taken as the first working nodes. If the amount of data that can be migrated of a working node in the first a working nodes is less than the data volume of each sub-data, the control node 310 divides the data to be migrated into b parts, thereby obtaining the data volume of each sub-data in this case. Repeat the above steps until the amount of data that can be migrated of each working node is greater than or equal to the data volume of each sub-data, thereby determining multiple first working nodes ranked in the front.

[0174] Among them, the above a and b are positive integers, and a is greater than or equal to 1, and b is greater than a.

[0175] In a possible embodiment, the data to be migrated is valid data determined by the control node 310 from the source storage cluster.

[0176] Since the data indicated by the user is data corresponding to a continuous section of addresses, the data indicated by the user contains both valid data and invalid data.

[0177] In a possible implementation, the control node 310 obtains a bitmap of the source storage cluster, and then uses the valid data in the source storage cluster as the data to be migrated according to the bitmap.

[0178] In a possible example, the control node 310 uses valid data among all data in the source storage cluster as data to be migrated.

[0179] In another possible example, the control node 310 uses valid data in a corresponding storage block / sector / cloud hard disk in the source storage cluster as data to be migrated according to the storage block / sector / cloud hard disk ID carried in the data migration request.

[0180] A bitmap is a data structure that uses binary bits (i.e., 0 or 1) to represent the state of a specific position. For example, 1 indicates that the corresponding data block is valid data or 0 indicates that the corresponding data block is invalid data, or 1 indicates that the data in the corresponding data block is available or 0 indicates that the data in the corresponding data block is unavailable.

[0181] In a bitmap, each data item (or data area) has a corresponding bit that can be set to 0 or 1. Usually, the bitmap is stored in the memory in the form of a binary number. Since each bit has only two states, a binary number can be used to represent its state, so the bitmap usually takes up very little storage space.

[0182] For example, since the amount of data indicated by the user is large, the data indicated by the user may correspond to multiple bits in the bitmap. The control node 310 queries the multiple bits corresponding to the data indicated by the user in the bitmap, and then determines the valid data in the data indicated by the user to obtain the data to be migrated.

[0183] In the present application, only valid data in the data indicated by the user, i.e., data to be migrated, is transferred, which reduces the amount of data to be migrated, thereby speeding up the migration speed and improving the efficiency of data migration. It also avoids the problem of migrating invalid data to the target storage cluster 240 and wasting the storage resources of the target storage cluster 240, thereby improving the utilization rate of the storage space in the target storage cluster.

[0184] In a possible embodiment, a method for a user to access target data in the data to be migrated during the above data migration is shown. Figure 5 As shown, Figure 5 A flow chart of a data access method provided by the present application, in which the user's input / output load has been mounted to the target storage cluster in this embodiment, and then, the data access request corresponding to the business will access the target storage cluster. Take the second working node in the working node cluster to execute the data access method as an example for explanation, and the second working node may be working node 500. The data access method may include the following steps S510 to S540.

[0185] Step S510 : The working node 500 obtains a snapshot bitmap of the target storage cluster 240 .

[0186] The snapshot bitmap of the target storage cluster 240 records the changes of files or data blocks in the target storage cluster 240, such as the newly stored / created data in the storage space, whether the data is modified, and other information.

[0187] In a possible implementation, the control node 310 reads the snapshot bitmap in the metadata of the target storage cluster 240. The metadata is stored in a designated area in the target storage cluster 240, and the control node 310 can obtain the snapshot bitmap by accessing the designated area.

[0188] Exemplarily, the snapshot bitmap of the target storage cluster 240 may be 00000000: 11010110, 00000008: 11111111. 00000000 and 00000008 represent the addresses of the data blocks or data areas, and each value in 11010110 and 11111111 represents the status of the corresponding sub-data blocks. 1 represents that the data block or sector is modified or occupied, and 0 represents that the data block or sector is not occupied. For example, the first 1 in 11010110 represents that the first data block at the address starting from 00000008 is modified.

[0189] In a possible scenario, the control node 310 determines the working node 500 from the multiple working nodes to ensure load balancing of the multiple working nodes.

[0190] Step S520 : The working node 500 determines whether the target data is stored in the target storage cluster 240 according to the snapshot bitmap.

[0191] In a possible implementation, the working node 500 determines whether the target data is stored in the second storage space according to the snapshot bitmap, including: the working node 500 determines whether there is a corresponding record in the snapshot bitmap according to the address (logical address) of the target data.

[0192] The working node 500 determines the bit in the snapshot bitmap corresponding to the address of the target data. If the bit is 1, it means that the target data is stored in the source storage cluster 230; if the bit is 0, it means that the target data is not stored in the target storage cluster 240, that is, the target data has not been migrated to the target storage cluster 240 and is still stored in the source storage cluster 230.

[0193] Exemplarily, the working node 500 may convert the address of the target data into a data block number, and then query the bit corresponding to the specified data block number in the snapshot bitmap.

[0194] For example, the address of the target data is 0x100, and the control node 310 converts the address 0x100 into a data block number of 2, and then checks whether the third bit in 11010110 is 1. If the third bit is 1, it indicates that the target data is stored in the target storage cluster 240; if the third bit is 0, it indicates that the target data is not stored in the target storage cluster 240, that is, the target data is stored in the source storage cluster 230.

[0195] In a possible scenario, the address of the target data in the target storage cluster 240 is obtained by converting the name of the target data into the corresponding address using an address conversion system by the control node 310. The address conversion system indicates the correspondence between the name of the data and the address.

[0196] Step S530 : If the target data is stored in the target storage cluster 240 , the working node 500 accesses the target data stored in the target storage cluster 240 .

[0197] If the record corresponding to the name or address of the target data in the snapshot bitmap is 1 (indicating newly created data or modified data), the working node 500 accesses the target data stored in the target storage cluster 240 .

[0198] The above data access request carries the logical address of the target data in the target storage cluster 240 , and the second working node obtains, modifies or updates the target data according to the logical address of the target data in the target storage cluster 240 .

[0199] Step S540 : If the target data is stored in the source storage cluster 230 , the working node 500 accesses the target data stored in the source storage cluster 230 .

[0200] If the record corresponding to the name or address of the target data in the snapshot bitmap is 0 (the data block is not occupied), it means that the target data has not been migrated to the target storage cluster 240 and is still stored in the source storage cluster 230 , and the working node 500 accesses the target data stored in the source storage cluster 230 .

[0201] During the data migration process, if the user needs to access part of the data to be migrated (target data), the control node 310 can determine whether the target data is stored in the source storage cluster 230 or the target storage cluster 240 by querying the snapshot bitmap, and then access the target data in the corresponding storage space. This avoids the problem of data disorder caused by the inconsistency between the actually accessed data and the target data due to unclear storage location of the target data, improves the accuracy of accessing the target data during the data migration process, and does not affect the normal operation of the business, that is, hot migration is achieved.

[0202] In a possible example, the snapshot bitmap records the changes of data blocks of a set granularity, such as a set granularity of 1M. The working node 500 reads the record in the snapshot bitmap corresponding to the target data at a granularity of 1M to determine whether the 1M data has been stored in the target storage cluster 240. If the 1M data has a corresponding record in the snapshot bitmap, that is, the 1M data is stored in the target storage cluster 240, the control node 310 obtains, updates, modifies or deletes the 1M data according to the address corresponding to the 1M data.

[0203] If the 1M data has no corresponding record in the snapshot bitmap, that is, the 1M data is still stored in the source storage cluster 230 , the working node 500 obtains, updates, deletes or modifies the 1M data according to the address of the 1M data in the source storage cluster 230 .

[0204] Repeat the above steps with a granularity of 1M until all the target data is read.

[0205] It is worth noting that the above-mentioned setting granularity can be adjusted as needed, such as the setting granularity can also be 2M or 512K.

[0206] In a possible embodiment, if the snapshot bitmap indicates that the target data is not stored in the target storage cluster 240 , the data migration method further includes: the working node 500 migrates the target data from the source storage cluster 230 to the target storage cluster 240 .

[0207] In a possible implementation, the working node 500 migrates the target data from the source storage cluster 230 to the target storage cluster 240 , including: the working node 500 migrates the data in the corresponding logical address to the target storage cluster 240 according to the logical address of the target data in the source storage cluster.

[0208] For the description of the logical address, reference can be made to the above-mentioned control node 310 sending the content of the logical address in the logical address matching each first working node to each first working node, which is not described in detail here. And for the content of the working node 500 migrating the data in the corresponding logical address to the target storage cluster 240, reference can be made to the above-mentioned description of the working node 330 performing data migration, which is not described in detail here.

[0209] In the present application, the second working node migrates the target data stored in the source storage cluster to the target storage cluster according to the user's access requirements (the user accesses the target data in the data to be migrated), thereby realizing migration according to the user's requirements.

[0210] After the working node 500 accesses the target data stored in the source storage cluster 230, the working node 500 migrates the data to be accessed from the storage space to the target storage cluster 240, so that the target data can be migrated from the source storage cluster 230 to the target storage cluster 240 by only obtaining the address of the target data once. This avoids the step of obtaining the address of the target data again to migrate the target data to the target storage cluster 240, thereby improving the migration speed.

[0211] Combined with the above Figures 1 to 5 , describes in detail the data migration method provided by the present application. In the following embodiments, the present application also provides a data migration system. Figure 6 As shown, Figure 6 The data migration system 600 includes a control node 610 and a working node 620. The data migration system is used to migrate data stored in a source storage cluster to a target storage cluster. Figure 6 Only one working node 620 among the multiple working nodes included in the data migration system 600 is shown.

[0212] The data migration system includes: a control node 610, which is used to obtain the data volume of the data to be migrated in the source storage cluster.

[0213] The control node 610 is further used to determine N first working nodes from multiple working nodes according to the amount of data to be migrated, wherein the first working node is a working node that meets a preset condition, and N is a positive integer greater than 1.

[0214] The control node 610 is further used to send data migration instructions to the N first working nodes.

[0215] The N first working nodes are used to migrate the data to be migrated from the source storage cluster to the target storage cluster according to the data migration instruction.

[0216] In a possible implementation manner, the control node 610 is specifically configured to: obtain a data migration request from a user, where the data migration request includes an identifier of a source storage cluster and / or an identifier of a target storage cluster.

[0217] In one possible implementation, the control node 610 is specifically used to: calculate the amount of migratable data of multiple working nodes per unit time, sort the multiple working nodes according to the amount of migratable data of the multiple working nodes, and then select the first N working nodes as the first working nodes according to the sorting results.

[0218] In a possible implementation, the control node 610 is also specifically used to calculate the amount of data that can be migrated by multiple working nodes per unit time according to the bandwidth, historical migration information and migration task information of the multiple working nodes, wherein the historical migration information is used to indicate the failure or success information of the multiple working nodes in executing historical migration tasks, and the migration task information is used to indicate the migration start time of the data to be migrated.

[0219] In a possible implementation, the control node 610 is specifically configured to obtain a bitmap of the source storage cluster, and use valid data in the source storage cluster as data to be migrated according to the bitmap. The bitmap is used to indicate whether the data stored in the source storage cluster is valid data.

[0220] In one possible implementation, during data migration, if a user accesses target data in the data to be migrated, the user's input / output load is mounted on the target storage cluster, and the second working node among the multiple working nodes is specifically used to: obtain a snapshot bitmap of the target storage cluster, and determine whether the target data is stored in the target storage cluster based on the snapshot bitmap.

[0221] The second working node is used to read the target data from the target storage cluster if the target data is stored in the target storage cluster.

[0222] The second working node is used to read the target data from the source storage cluster if the target data is stored in the source storage cluster, and migrate the target data to the target storage cluster.

[0223] For more detailed functions implemented by the control node 610 and the working node 620 in the data migration system, please refer to the above Figures 3 to 5 The contents shown will not be repeated here.

[0224] As an example of a software functional unit, a control node may include code running on a computing instance. The computing instance may be at least one of a physical host (computing device), a virtual machine, a container, and other computing devices. Furthermore, the computing device may be one or more. For example, a control node may include code running on multiple hosts / virtual machines / containers. It should be noted that multiple hosts / virtual machines / containers for running the application may be distributed in the same region or in different regions. Multiple hosts / virtual machines / containers for running the code may be distributed in the same AZ or in different AZs, each AZ including one data center or multiple data centers with close geographical locations. Typically, a region may include multiple AZs.

[0225] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same VPC or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway must be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0226] As an example of a hardware functional unit, the control node may include at least one computing device, such as a server, etc. Alternatively, the control node may also be a device implemented using ASIC or PLD, etc. The PLD may be implemented using CPLD, FPGA, GAL or any combination thereof.

[0227] The multiple computing devices included in the control node can be distributed in the same region or in different regions. The multiple computing devices included in the control node can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the control node can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0228] It is worth noting that the control node 310 in the aforementioned embodiment may correspond to the control node 610, the first working node may correspond to the working node 620, and may correspond to the method for executing the embodiment of the present application. Figures 3 to 5The corresponding corresponding subjects, and the operations and / or functions of each node in the data migration system 600 are respectively to achieve Figures 3 to 5 For the sake of brevity, the corresponding processes of each method in the corresponding embodiments are not repeated here.

[0229] in addition, Figure 6 The data migration system 600 shown can also be implemented through a communication device, where the communication device may refer to the control node 310 or the first working node in the aforementioned embodiment, or, when the communication device is a chip or a chip system applied to a processing device, the data migration system can also be implemented through the chip or the chip system.

[0230] An embodiment of the present application also provides a chip system, which includes a control circuit and an interface circuit. The interface circuit is used to obtain the data volume of the data to be migrated, and the control circuit is used to implement the function of the control node in the above method according to the data volume of the data to be migrated.

[0231] In a possible design, the chip system further includes a memory for storing program instructions and / or data. The chip system may be composed of a chip, or may include a chip and other discrete devices.

[0232] The present application also provides a computing device. Figure 7 As shown, Figure 7 The present application provides a schematic diagram of a computing device structure, wherein the computing device 700 includes: a bus 702, a processor 704, a memory 706, and a communication interface 708. The processor 704, the memory 706, and the communication interface 708 communicate with each other via the bus 702. The computing device 700 may be a server or a terminal device, and the computing device 700 may be the control node or the working node mentioned above. It is worth noting that the present application does not limit the number of processors and memories in the computing device 700.

[0233] The bus 702 may be, but is not limited to: a PCIe bus, a USB, or an inter-integrated circuit (I2C), an EISA (extended industry standard architecture) bus, UB, CXL, CCIX, etc. The bus 702 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 The bus 702 may include a path for transmitting information between various components of the computing device 700 (eg, the memory 706, the processor 704, and the communication interface 708).

[0234] The processor 704 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0235] The memory 706 may include a volatile memory, such as a random access memory (RAM). The memory 706 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0236] The memory 706 stores executable program codes, and the processor 704 executes the executable program codes to respectively implement the functions of the aforementioned control node 610 and the working node 620, thereby implementing the aforementioned data migration method. That is, the memory 706 stores instructions for executing the data migration method.

[0237] The communication interface 708 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 700 and other devices or communication networks.

[0238] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device, which can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0239] like Figure 8 As shown, Figure 8 A schematic diagram of the structure of a computer cluster provided in the present application. The computing device cluster includes at least one computing device 700. The memory 706 in one or more computing devices 700 in the computing device cluster may store the same instructions for executing the data migration method. The computer cluster may be the above-mentioned data migration system.

[0240] In some possible implementations, the memory 706 of one or more computing devices 700 in the computing device cluster may also store partial instructions for executing the data migration method. In other words, the combination of one or more computing devices 700 may jointly execute instructions for executing the data migration method.

[0241] It should be noted that the memory 706 in different computing devices 700 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the data migration system. That is, the instructions stored in the memory 706 in different computing devices 700 can implement the functions of one or more nodes in the control node 610 and the working node 620.

[0242] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig. 9 A possible implementation is shown. Fig. 9 As shown, Fig. 9 A schematic diagram of a connection between computing devices provided in the present application, wherein two computing devices 700A and 700B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this possible implementation, the memory 706 in the computing device 700A stores instructions for executing the functions of the control node 610. At the same time, the memory 706 in the computing device 700B stores instructions for executing the functions of the above-mentioned working node 620.

[0243] Figure 8 The connection method between the computing device clusters shown may be based on the fact that the present application requires multiple working nodes to execute the data migration process, and therefore it is considered that the functions implemented by the working node 620 are handed over to the computing device 700B for execution.

[0244] It should be understood that Figure 8 The functions of the computing device 700A shown in FIG. 7 may also be completed by multiple computing devices 700. Similarly, the functions of the computing device 700B may also be completed by multiple computing devices 700.

[0245] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the data migration method.

[0246] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by the computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the data migration method.

[0247] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present application is executed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device or other programmable device. The computer program or instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instruction may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, for example, a floppy disk, a hard disk, a tape; it may also be an optical medium, for example, a digital video disc (DVD); it may also be a semiconductor medium, for example, a solid state drive (SSD).

[0248] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A data migration method, characterized in that: The method is applied to a data migration system, the data migration system is used to migrate data stored in a source storage cluster to a target storage cluster, the data migration system includes a control node and multiple working nodes, and the method includes: The control node obtains the amount of data to be migrated in the source storage cluster; The control node determines, according to the amount of the data to be migrated, N first working nodes from the multiple working nodes, wherein the first working nodes are working nodes that meet a preset condition, and N is a positive integer greater than 1; The control node sends a data migration instruction to the N first working nodes; The N first working nodes migrate the data to be migrated from the source storage cluster to the target storage cluster according to the data migration instruction.

2. The method according to claim 1, characterized in that The method further comprises: The control node obtains a data migration request from a user, where the data migration request includes an identifier of the source storage cluster and / or an identifier of the target storage cluster.

3. The method according to claim 1 or 2, characterized in that: The control node determines N first working nodes from the multiple working nodes, including: The control node calculates the amount of data that can be migrated by the multiple working nodes within a unit time; The control node sorts the multiple working nodes according to the amount of migrateable data of the multiple working nodes; The control node selects the first N working nodes as the first working nodes according to the sorting result.

4. The method according to claim 3, characterized in that The control node calculates the amount of data that can be migrated by the multiple working nodes per unit time, including: The control node calculates the amount of data that can be migrated by the multiple working nodes per unit time according to the bandwidth, historical migration information and migration task information of the multiple working nodes, wherein the historical migration information is used to indicate the failure or success information of the multiple working nodes in executing historical migration tasks, and the migration task information is used to indicate the migration start time of the data to be migrated.

5. The method according to any one of claims 1 to 4, characterized in that: The control node obtains the amount of data to be migrated in the source storage cluster, including: The control node obtains a bitmap of the source storage cluster, where the bitmap is used to indicate whether the data stored in the source storage cluster is valid data; The control node uses the valid data in the source storage cluster as data to be migrated according to the bitmap.

6. The method according to any one of claims 1 to 5, characterized in that: During data migration, if a user accesses target data in the data to be migrated, the user's input / output load is mounted on the target storage cluster, and the method further includes: The second working node obtains a snapshot bitmap of the target storage cluster; The second working node determines whether the target data is stored in the target storage cluster according to the snapshot bitmap; If the target data is stored in the target storage cluster, the second working node reads the target data from the target storage cluster; If the target data is stored in the source storage cluster, the second working node reads the target data from the source storage cluster and migrates the target data to the target storage cluster.

7. A data migration system, characterized in that: The data migration system is used to migrate data stored in a source storage cluster to a target storage cluster. The data migration system includes a control node and multiple working nodes. The data migration system includes: The control node is used to obtain the data volume of the data to be migrated in the source storage cluster; The control node is further configured to determine N first working nodes from the multiple working nodes according to the amount of data to be migrated, wherein the first working nodes are working nodes that meet a preset condition, and N is a positive integer greater than 1; The control node is further used to send a data migration instruction to the N first working nodes; The N first working nodes are used to migrate the data to be migrated from the source storage cluster to the target storage cluster according to the data migration instruction.

8. The data migration system according to claim 7, characterized in that: The control node is specifically used for: A data migration request of a user is obtained, where the data migration request includes an identifier of the source storage cluster and / or an identifier of the target storage cluster.

9. The data migration system according to claim 7 or 8, characterized in that: The control node is specifically used for: The amount of migrateable data of the multiple working nodes per unit time is calculated, and the multiple working nodes are sorted according to the amount of migrateable data of the multiple working nodes, and then the first N working nodes are selected as the first working nodes according to the sorting result.

10. The data migration system according to claim 9, characterized in that: The control node is further specifically used for: According to the bandwidth, historical migration information and migration task information of the multiple working nodes, the amount of data that can be migrated by the multiple working nodes per unit time is calculated, wherein the historical migration information is used to indicate the failure or success information of the multiple working nodes in executing historical migration tasks, and the migration task information is used to indicate the migration start time of the data to be migrated.

11. The data migration system according to any one of claims 7 to 10, characterized in that: The control node is specifically used for: A bitmap of the source storage cluster is obtained, and according to the bitmap, valid data in the source storage cluster is used as data to be migrated; the bitmap is used to indicate whether the data stored in the source storage cluster is valid data.

12. The data migration system according to any one of claims 7 to 11, characterized in that: During data migration, if a user accesses target data in the data to be migrated, the user's input / output load is mounted on the target storage cluster, and a second working node among the plurality of working nodes is specifically used to: Acquire a snapshot bitmap of the target storage cluster, and determine whether the target data is stored in the target storage cluster according to the snapshot bitmap; The second working node is configured to read the target data from the target storage cluster if the target data is stored in the target storage cluster; The second working node is used to read the target data from the source storage cluster and migrate the target data to the target storage cluster if the target data is stored in the source storage cluster.

13. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method according to any one of claims 1 to 6.

15. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 6.

Images