Data migration method and device, electronic equipment and storage medium

Abstract

The invention provides a data migration method and device, electronic equipment and a storage medium. The distributed storage system comprises a scheduling node, a routing node and a data storage cluster, the data storage cluster comprises a source data storage cluster and a target data storage cluster, and the method comprises the following steps: in response to a data query request sent by the scheduling node, sending the data query request to a query node in each data storage cluster; receiving feedback information returned by the query node for the data query request; based on the feedback information, determining a data storage cluster storing the to-be-migrated data as a source data storage cluster, and determining a storage node storing the to-be-migrated data in the source data storage cluster as a target storage node; and the node identifier of the target storage node is sent to the scheduling node, and the node identifier is used for triggering the scheduling node to migrate the to-be-migrated data from the source data storage cluster to the target data storage cluster. According to the invention, the influence of data migration on the data service can be effectively reduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data migration method, device, electronic device and storage medium. Background Art

[0002] Cluster technology is a technology that has emerged recently with the development of the Internet. It connects a group of independent computers through a high-speed network to form a cluster system that can be managed and scheduled in a unified manner. It is used to handle services in various big data scenarios, thereby improving the scalability and reliability of services. In order to ensure that services in big data scenarios can run normally, cluster systems usually store massive amounts of data to support various services in big data scenarios.

[0003] In the related art, data stored in a cluster system needs to be migrated to another cluster system. When performing data migration, it is generally necessary to suspend the big data business of the cluster system, resulting in the inability to respond to data query requests and perform data queries. The business can only be resumed on the migrated cluster system after all data has been migrated. Since there is a lot of data to be migrated in the big data scenario, the business will be suspended for a long time, resulting in a greater impact of data migration on the data business. Summary of the invention

[0004] The embodiments of the present application provide a data migration method, device, electronic device, computer-readable storage medium, and computer program product, which can effectively reduce the impact of data migration on data services.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The distributed storage system includes: a scheduling node, a routing node and a data storage cluster, the data storage cluster includes a query node and a storage node, and the data storage cluster includes a source data storage cluster and a target data storage cluster.

[0007] An embodiment of the present application provides a data migration method, comprising: in response to a data query request sent by the scheduling node, sending a data query request to a query node in the data storage cluster, the data query request being used to query whether the data to be migrated is stored in the data storage cluster; receiving feedback information returned by the query node in response to the data query request, wherein the feedback information is used to indicate whether the data to be migrated is stored in the data storage cluster; based on the feedback information, determining that the data storage cluster storing the data to be migrated is a source data storage cluster, and determining that the storage node storing the data to be migrated in the source data storage cluster is a target storage node; and sending a node identifier of the target storage node to the scheduling node, so that the scheduling node migrates the data to be migrated from the source data storage cluster to the target data storage cluster based on the scheduling node.

[0008] An embodiment of the present application provides a data migration device, wherein the distributed storage system includes: a request sending module, configured to send the data query request to the query node in the data storage cluster in response to the data query request sent by the scheduling node, wherein the data query request is used to query whether the data to be migrated is stored in the data storage cluster, the data storage cluster includes a query node and a storage node, and the multiple data storage clusters include a target data storage cluster; a receiving module, configured to receive feedback information returned by each of the query nodes in response to the data query request, wherein the feedback information is used to indicate whether the data to be migrated is stored in the corresponding data storage cluster; a determining module, configured to determine, based on the feedback information, that the data storage cluster storing the data to be migrated is the source data storage cluster, and to determine that the storage node storing the data to be migrated in the source data storage cluster is the target storage node; an identification sending module, configured to send the node identification of the target storage node to the scheduling node, so that the scheduling node migrates the data to be migrated from the source data storage cluster to the target data storage cluster based on the node identification.

[0009] In the above scheme, the above determination module is also used to compare the storage address with the address range of the storage nodes in the source data storage cluster to obtain a first comparison result when the feedback information carries the storage address of the data to be migrated; based on the first comparison result, determine the target storage node from the storage nodes in the source data storage cluster; when the feedback information does not carry the storage address of the data to be migrated, send a node query request to the source data storage cluster, and the node query request is used to query the target storage node among the storage nodes in the source data storage cluster; receive the node identifier of the target storage node storing the data to be migrated returned by the source data storage cluster in response to the node query request.

[0010] In the above scheme, the above determination module is further used to determine the corresponding storage node as the target storage node storing the data to be migrated when the first comparison result indicates that the storage address is within the address range of the corresponding storage node.

[0011] In the above scheme, the above data migration device also includes: a reading module, which is used to forward the data reading request for the data to be migrated sent by the scheduling node to the query node in the source data storage cluster, and the data reading request is used to trigger the query node to read the data to be migrated from the target storage node; a data receiving module is used to receive the data to be migrated read by the query node in response to the data reading request, and send the received data to be migrated to the target data storage cluster.

[0012] In the above scheme, the above data migration device also includes: a target node module, which is used to send a storage space query request to the query node in the target data storage cluster, and the storage space query request is used to trigger the query node to query the remaining storage space of each storage node in the target data storage cluster; receive the remaining storage space of the storage node returned by the query node in response to the storage space query request, and determine the reference storage node from the storage nodes of the target data storage cluster in combination with the remaining storage space and the data volume of the data to be migrated; send the target node identifier of the reference storage node to the query node in the target data storage cluster, and the target node identifier is used to trigger the query node to send the data to be migrated to the reference storage node after receiving the data to be migrated; the above target node module is also used to send the received data to be migrated to the query node in the target data storage cluster, so that the query node creates an index entry for the data to be migrated and sends the data to be migrated to the reference storage node.

[0013] In the above scheme, the above target node module is also used to compare the remaining storage space with the data volume to obtain a second comparison result; when the second comparison result indicates that the remaining storage space can accommodate the data volume to be migrated, the storage node is determined as a candidate storage node; when the number of the candidate storage nodes is one, the candidate storage node is determined as the reference storage node; when the number of the candidate storage nodes is multiple, the candidate storage node with the smallest remaining storage space is determined as the reference storage node.

[0014] In the above scheme, the above data migration device also includes: a deletion module, which is used to send a data deletion request to the query node in the source data storage cluster; the data deletion request is used to trigger the query node to delete the index entry of the data to be migrated stored in the query node, and send a data deletion request to the target storage node in the source data storage cluster, so that the target storage node deletes the data to be migrated stored in the target storage node; receiving the indication information returned by the query node in the source data storage cluster in response to the data deletion request, the indication information is used to trigger the routing node to send the received data to be migrated to the target data storage cluster; the above target node module is also used to send the data to be migrated to the target data storage cluster in response to the received indication information.

[0015] In the above scheme, the above-mentioned target node module is also used to send the data to be migrated to the target data storage cluster in response to receiving the data to be migrated; the above-mentioned deletion module is also used to send a data deletion request to the query node in the source data storage cluster; wherein the data deletion request is used to trigger the query node to delete the index entry of the data to be migrated stored in the query node, and send a data deletion request to the target storage node in the source data storage cluster, so that the target storage node deletes the data to be migrated stored in the target storage node; receive the indication information returned by the query node in the source data storage cluster in response to the data deletion request, and the indication information is used to indicate that the data to be migrated no longer exists in the source data storage cluster.

[0016] In the above scheme, the above data migration device also includes: a query module, used to send a storage space query request to the query node in the data storage cluster, and the storage space query request is used to trigger the query node to query the overall remaining storage space of the storage nodes in the target data storage cluster; receive the overall remaining storage space returned by the data storage cluster in response to the storage space query request; based on the overall remaining storage space, determine the target data storage cluster from the multiple data storage clusters.

[0017] In the above scheme, the above query module is also used to perform the following processing on the overall remaining storage space of the data storage cluster: compare the overall remaining storage space with the data volume of the data to be migrated to obtain a third comparison result; when the third comparison result indicates that the overall remaining storage space can accommodate the data volume of the data to be migrated, determine the data storage cluster as a candidate data storage cluster; when the number of the candidate data storage clusters is one, determine the candidate data storage cluster as the target data storage cluster; when the number of the candidate data storage clusters is multiple, determine the candidate data storage cluster with the smallest overall remaining storage space as the target data storage cluster.

[0018] In the above scheme, the above data migration device also includes: a reading module, which is used to respond to the data reading request for the data to be migrated sent by the scheduling node, forward the data reading request to the query node in the source data storage cluster, and the data reading request is used to trigger the query node to read the data to be migrated from the target storage node; receive the data to be migrated read by the query node in response to the data reading request; and send the data to be migrated to the scheduling node.

[0019] In the above scheme, the reading module is also used to forward the data reading request to the query node in the source data storage cluster when there are multiple source data storage clusters, or to forward the data reading request to any query node in the source data storage cluster; when there is one source data storage cluster, forward the data reading request to the query node in the source data storage cluster; when receiving the data to be migrated read by multiple query nodes in response to the data reading request, deduplicate the multiple copies of the data to be migrated to obtain the target data to be migrated; and send the target data to be migrated to the scheduling node.

[0020] In the above solution, the reading module is further used to send the received data to be migrated to the scheduling node in response to a data reading request for the data to be migrated sent by the scheduling node.

[0021] In the above scheme, the reading module is also used to respond to the data reading request for the data to be migrated sent by the scheduling node, and forward the data reading request to the query node in the target data storage cluster, wherein the data reading request is used to trigger the query node to read the data to be migrated from the target data storage cluster; receive the data to be migrated read by the query node in response to the data reading request; and send the data to be migrated to the scheduling node.

[0022] In the above scheme, the reading module is also used to respond to the data reading request for the data to be migrated sent by the scheduling node, forward the data reading request to the source data storage cluster or the target data storage cluster; receive the data to be migrated read by the source data storage cluster or the target data storage cluster; and send the data to be migrated to the scheduling node.

[0023] In the above scheme, the reading module is also used to respond to the data reading request for the data to be migrated sent by the scheduling node, forward the data reading request to the source data storage cluster and the target data storage cluster; receive the data to be migrated returned in response to the data reading request, and when the data to be migrated returned by the source data storage cluster and the data to be migrated returned by the target data storage cluster are received, compare the data to be migrated returned by the source data storage cluster and the target data storage cluster to obtain a comparison result; when the comparison result indicates that the data to be migrated returned by the source data storage cluster and the target data storage cluster are the same, deduplicate the returned data to be migrated to obtain the target data to be migrated; and send the target data to be migrated to the scheduling node.

[0024] An embodiment of the present application provides a data migration device, which is applied to a scheduling node of a distributed storage system, wherein the distributed storage system includes: the scheduling node, a routing node and a data storage cluster, wherein the data storage cluster includes a source data storage cluster and a target data storage cluster, and the device includes: a request sending module, which is used to send a data query request to the routing node, so that the routing node sends the data query request to the query node in the data storage cluster in response to the data query request, and receives feedback information returned by the query node for the data query request, and based on the feedback information, determines that the data storage cluster storing the data to be migrated is the source data storage cluster, and determines that the storage node storing the data to be migrated in the source data storage cluster is the target storage node; wherein the data query request is used to query whether the data storage cluster stores the data to be migrated, and the data storage cluster includes the query node and a storage node; and a response module, which is used to migrate the data to be migrated from the source data storage cluster to the target data storage cluster in response to the node identifier of the target storage node sent by the routing node.

[0025] In the above scheme, the above response module is also used to send a data migration request for the data to be migrated to the routing node in response to the node identifier of the target storage node sent by the routing node, so that the routing node forwards the data migration request to the query node in the source data storage cluster, and receives the data to be migrated read by the query node for the data migration request, and sends the received data to be migrated to the target data storage cluster; wherein, the data migration request is used to trigger the query node to read the data to be migrated from the target storage node.

[0026] The embodiment of the present application provides a data migration device, which is applied to a distributed storage system. The distributed storage system includes: a scheduling node, a routing node and a data storage cluster, wherein the data storage cluster includes a source data storage cluster and a target data storage cluster. The device includes: a scheduling sending module, which is used for the scheduling node to send a data query request to the routing node, wherein the data query request is used to query whether the data storage cluster stores data to be migrated; a routing response module, which is used for the routing node to send the data query request to the query node in the data storage cluster in response to the data query request, and receive feedback information returned by the query node for the data query request, wherein the data storage cluster includes the query node and a storage node; a routing determination module, which is used for the routing node to determine, based on each piece of feedback information, that the data storage cluster storing the data to be migrated is the source data storage cluster, and to determine that the storage node storing the data to be migrated in the source data storage cluster is the target storage node; a routing sending module, which is used for the routing node to send the node identifier of the target storage node to the scheduling node; and a scheduling receiving module, which is used for the scheduling node to receive the node identifier of the target storage node, and migrate the data to be migrated from the source data storage cluster to the target data storage cluster based on the node identifier.

[0027] An embodiment of the present application provides an electronic device, including:

[0028] A memory for storing computer executable instructions or computer programs;

[0029] The processor is used to implement the data migration method provided in the embodiment of the present application when executing the computer executable instructions or computer programs stored in the memory.

[0030] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for causing a processor to execute and implement the data migration method provided in the embodiment of the present application.

[0031] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the data migration method described in the embodiment of the present application.

[0032] The embodiments of the present application have the following beneficial effects:

[0033] In response to the data query request for the data to be migrated sent by the scheduling node, the routing node sends a data query request to the query node in the data storage cluster, and receives feedback information returned by the query node for the data query request. Based on the feedback information, a source data storage cluster storing the data to be migrated and a target storage node storing the data to be migrated in the source data storage cluster are determined from multiple data storage clusters, and a node identifier of the target storage node is sent to the scheduling node, so that the scheduling node migrates the data to be migrated from the source data storage cluster to the target data storage cluster. In this way, the source data storage cluster storing the data to be migrated and the target storage node storing the data to be migrated in the source data storage cluster are determined through the routing node, and the node identifier of the target storage node is sent to the scheduling node through the routing node to migrate the data to be migrated from the source data storage cluster to the target data storage cluster. The distributed storage system always stores the data to be migrated, so that when data is migrated between different data storage clusters in the distributed storage system, the distributed storage system can also respond normally to the data reading of the scheduling node for the data to be migrated without interrupting the business, so that the data to be migrated is read from the distributed storage system, that is, the process of data migration between different data storage clusters in the distributed storage system will not interfere with the data reading business, so that the data reading business can operate normally, and data migration is realized without the data reading business being aware of it, thereby effectively reducing the impact of data migration on the data reading business. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of the architecture of the data migration system provided in an embodiment of the present application;

[0035] Figures 2 to 4 is a schematic diagram of the structure of an electronic device for performing data migration in a distributed storage system provided in an embodiment of the present application;

[0036] Figure 5 It is a schematic diagram of the principle of a distributed storage system provided in an embodiment of the present application;

[0037] Figure 6-Figure 10It is a flowchart of a data migration method provided in an embodiment of the present application;

[0038] Fig.11 It is a schematic diagram of the principle of a distributed storage system provided in an embodiment of the present application;

[0039] Figure 12 to Figure 13 It is a flowchart of a data migration method provided in an embodiment of the present application;

[0040] Fig.14 It is a principle schematic diagram of an application scenario of the data migration method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0042] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0043] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0045] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0046] 1) Distributed File System (Hadoop Distributed File System, HDFS): refers to a distributed file system designed to run on commodity hardware. It has many similarities with existing distributed file systems, but at the same time, it is also very different from other distributed file systems. HDFS is a highly fault-tolerant system suitable for deployment on cheap machines. HDFS can provide high-throughput data access and is very suitable for applications on large data sets. HDFS relaxes some POSIX constraints to achieve the purpose of streaming file system data. HDFS has the characteristics of high fault tolerance and is designed to be deployed on low-cost hardware. It also provides high throughput to access application data and is suitable for applications with large data sets. HDFS relaxes POSIX requirements so that streaming access can be achieved to data in the file system.

[0047] 2) Data warehouse tool (Hive): It is a data warehouse tool based on Hadoop, used for data extraction, transformation, and loading. It is a mechanism for storing, querying, and analyzing large-scale data stored in Hadoop. The Hive data warehouse tool can map structured data files to a database table, and provides SQL query functions, which can convert SQL statements into MapReduce tasks for execution. The advantage of Hive is that it has a low learning cost and can implement fast MapReduce statistics through similar SQL statements, making MapReduce simpler without having to develop a dedicated MapReduce application. Hive is very suitable for statistical analysis of data warehouses.

[0048] 3) Client: An application running in a terminal to provide various services, such as a video player client, a game client, etc.

[0049] 4) In response to: used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed may be in real time or have a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed are executed.

[0050] 5) Cloud Technology: Cloud technology refers to a hosting technology that unifies hardware, software, network and other resources within a wide area network or local area network to achieve data computing, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool that can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of technical network systems require a large amount of computing and storage resources, such as video websites, image websites and more portal websites. With the rapid development and application of the Internet industry, in the future, each item may have its own identification mark, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately. All kinds of industry data require strong system backing support, which can only be achieved through cloud computing.

[0051] 6) Business data: refers to the data used by the business application to run, which can be understood as the direct object of the business application to call the database (such as adding, deleting, checking and modifying). Business data includes but is not limited to various business resource data required for business applications to respond to business requests, user business data collected and stored by business applications, etc. Specifically, business data may include text data, image data, file data, etc. In the embodiment of the present application, all data stored in the cluster system except metadata can be referred to as business data. Business data is usually accessed in the system in the form of database tables.

[0052] 7) Cluster system, also known as computer cluster, refers to a group of loosely integrated computer software and / or hardware connected together through a high-speed network, working closely together to complete the calculation, which is equivalent to using multiple computers to complete the work that one computer needs to perform. In a cluster system, a single computer can be regarded as a computing node to complete part of the computing task, and multiple computing computers work together to complete the entire computing task. Specifically, the cluster system may include a server cluster and / or a database cluster.

[0053] During the implementation of the embodiments of the present application, the applicant discovered that the related technology has the following problems:

[0054] In the related art, data stored in a cluster system needs to be migrated to another cluster system. When performing data migration, it is generally necessary to suspend the big data business of the cluster system, that is, the cluster system may not store the corresponding data when performing data migration, resulting in the inability to respond to data query requests and perform data queries. The business can only be resumed on the migrated cluster system after all data has been migrated. Since there is a lot of data to be migrated in the big data scenario, the business will be suspended for a long time, resulting in low data migration efficiency.

[0055] Embodiments of the present application provide a data migration method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can effectively reduce the impact of data migration on data services. The following describes an exemplary application of the data migration system provided in embodiments of the present application.

[0056] See also Figure 1 , Figure 1 It is a schematic diagram of the architecture of the data migration system 100 provided in an embodiment of the present application. The terminal (terminal 400 is shown as an example) is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0057] The terminal 400 is used for the user to use the client 410 to display the data to be migrated on the graphical interface 410-1 (graphic interface 410-1 is shown as an example). The terminal 400 and the server 200 are connected to each other via a wired or wireless network.

[0058] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart TV, a smart watch, a car terminal, etc., but is not limited thereto. The electronic device provided in the embodiment of the present application may be implemented as a terminal or as a server. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiment of the present application.

[0059] In some embodiments, a routing node in a distributed storage system running on server 200 sends a data query request to a query node in a data storage cluster in response to a data query request for data to be migrated sent by a scheduling node, receives feedback information returned by the query node in response to the data query request, and based on the feedback information, determines a source data storage cluster storing the data to be migrated and a target storage node in the source data storage cluster storing the data to be migrated from multiple data storage clusters, and sends a node identifier of the target storage node to the scheduling node, wherein the node identifier is used to trigger the scheduling node to migrate the data to be migrated from the source data storage cluster to the target data storage cluster.

[0060] In other embodiments, the routing node in the distributed storage system running on the terminal 400 sends a data query request to the query node in the data storage cluster in response to the data query request for the data to be migrated sent by the scheduling node, receives feedback information returned by the query node in response to the data query request, and based on the feedback information, determines a source data storage cluster storing the data to be migrated and a target storage node in the source data storage cluster storing the data to be migrated from multiple data storage clusters, and sends a node identifier of the target storage node to the scheduling node, wherein the node identifier is used to trigger the scheduling node to migrate the data to be migrated from the source data storage cluster to the target data storage cluster.

[0061] In other embodiments, the embodiments of the present application can be implemented with the aid of cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or a local area network to achieve data computing, storage, processing, and sharing.

[0062] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form a resource pool that can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources.

[0063] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device 500 for performing data migration in a distributed storage system provided in an embodiment of the present application, wherein: Figure 2 The electronic device 500 shown may be Figure 1 The server 200 or the terminal 400 in Figure 2 The electronic device 500 shown includes: at least one processor 410, a memory 450, and at least one network interface 420. The various components in the electronic device 500 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 440 is not described in detail. Figure 2 Various buses are labeled as bus system 440 .

[0064] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0065] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0066] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0067] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0068] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0069] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include: Bluetooth, Wireless Fidelity (WiFi), and Universal Serial Bus (USB).

[0070] In some embodiments, the data migration device provided in the embodiments of the present application can be implemented in a software manner. Figure 2 The data migration device 455 stored in the memory 450 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a request sending module 4551, a receiving module 4552, a determination module 4553, and an identification sending module 4554. These modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0071] See also Figure 3 , Figure 3 is a schematic diagram of the structure of an electronic device 600 for performing data migration in a distributed storage system provided in an embodiment of the present application, wherein: Figure 3 The electronic device 600 shown may be Figure 1 The server 200 or the terminal 400 in Figure 3The electronic device 600 shown includes: at least one processor 510, a memory 550, and at least one network interface 520. The various components in the electronic device 600 are coupled together via a bus system 540. It is understood that the bus system 540 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 540 is not described in detail. Figure 3 Various buses are labeled as bus system 540 .

[0072] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0073] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0074] The network communication module 552 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 520. Exemplary network interfaces 520 include: Bluetooth, Wireless Fidelity (WiFi), and Universal Serial Bus (USB).

[0075] In some embodiments, the data migration device provided in the embodiments of the present application can be implemented in a software manner. Figure 3 The data migration device 555 stored in the memory 550 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a request sending module 5551, a response module 5552, and these modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0076] See also Figure 4 , Figure 4 is a schematic diagram of the structure of an electronic device 700 for performing data migration in a distributed storage system provided in an embodiment of the present application, wherein: Figure 4 The electronic device 700 shown may be Figure 1 The server 200 or the terminal 400 in Figure 4The electronic device 700 shown includes: at least one processor 610, a memory 650, and at least one network interface 620. The various components in the electronic device 700 are coupled together via a bus system 640. It is understood that the bus system 640 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 640 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 640 is not described in detail. Figure 4 Various buses are labeled as bus system 640 .

[0077] In some embodiments, the memory 650 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0078] Operating system 651, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0079] The network communication module 652 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 620. Exemplary network interfaces 620 include: Bluetooth, Wireless Fidelity (WiFi), and Universal Serial Bus (USB).

[0080] In some embodiments, the data migration device provided in the embodiments of the present application can be implemented in a software manner. Figure 4 The data migration device 655 stored in the memory 650 is shown, which can be software in the form of a program and a plug-in, including the following software modules: a scheduling sending module 6551, a routing response module 6552, a routing determination module 6553, a routing sending module 6554, and a scheduling receiving module 6555. These modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0081] In other embodiments, the data migration device provided in the embodiments of the present application can be implemented in hardware. As an example, the data migration device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data migration method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or other electronic components.

[0082] In some embodiments, the terminal or server can implement the data migration method provided in the embodiment of the present application by running a computer program or a computer executable instruction. For example, the computer program can be a native program (e.g., a dedicated data migration program) or a software module in the operating system, for example, a data migration module that can be embedded in any program (such as an instant messaging client, an album program, an electronic map client, a navigation client); for example, it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run. In short, the above-mentioned computer program can be any form of application, module or plug-in.

[0083] The data migration method provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the server or terminal provided in the embodiment of the present application.

[0084] In some embodiments, see Figure 5 , Figure 5 Schematic diagram of the distributed storage system provided by the embodiment of the present application. The distributed storage system includes: a scheduling node, a routing node and a data storage cluster (for example, Figure 5 The data storage cluster 1 and the data storage cluster 2 are shown, the data storage cluster includes a query node and a storage node, and the data storage cluster includes a target data storage cluster and a source data storage cluster.

[0085] In some embodiments, the above-mentioned scheduling node can be implemented by a migration tool and an application, or can be implemented by a migration tool alone. The specific implementation method of the scheduling node does not constitute a limitation on the embodiments of the present application.

[0086] See also Figure 6 , Figure 6is a flow chart of the data migration method provided in the embodiment of the present application, which will be combined with Figure 6 Steps 101 to 104 are shown for illustration. The data migration method provided in the embodiment of the present application can be implemented by a server or a terminal alone, or by a server and a terminal in collaboration. The following will take the implementation by a server alone as an example to illustrate the data migration method provided in the embodiment of the present application from the perspective of a routing node.

[0087] In step 101, in response to a data query request sent by a scheduling node, a data query request is sent to a query node in a data storage cluster.

[0088] In some embodiments, the data query request is used to query whether the data storage cluster stores data to be migrated.

[0089] In some embodiments, after receiving a data query request, a query node in the data storage cluster queries whether the data storage cluster stores data to be migrated.

[0090] In some embodiments, in the above step 101, sending a data query request to the query nodes in the data storage cluster may be simultaneously sending the data query request to each query node in the data storage cluster, or may be sending the data query request to the query nodes in the data storage cluster in sequence. Below, the data migration method provided in the embodiments of the present application is explained for the above two situations respectively.

[0091] In some embodiments, when a data query request is sent to each query node in the data storage cluster at the same time, after receiving feedback information returned by each query node in response to the data query request, based on the feedback information returned by each query node, the data storage cluster storing the data to be migrated is determined to be the source data storage cluster, and the storage node storing the data to be migrated in the source data storage cluster is determined to be the target storage node.

[0092] As an example, see Figure 7 , Figure 7 Schematic diagram of the distributed storage system provided by the embodiment of the present application. The routing node 92 responds to the data query request for the data to be migrated sent by the scheduling node 91, and sends a query to the data storage cluster (for example, Figure 7 The query nodes in the data storage cluster 93, data storage cluster 94, and data storage cluster 95 shown in the figure send data query requests in parallel. After receiving the data query request, the query node in the data storage cluster 93 responds to the data query request and queries whether the data storage cluster 93 stores the data to be migrated. After receiving the data query request, the query node in the data storage cluster 94 responds to the data query request and queries whether the data storage cluster 93 stores the data to be migrated.

[0093] In this way, by simultaneously sending a data query request to each query node in the data storage cluster, each query node can query the feedback information in parallel, thereby effectively improving the query efficiency of the feedback information and thus effectively improving the data migration efficiency.

[0094] In other embodiments, when sending data query requests to the query nodes in the data storage cluster in sequence, after receiving feedback information returned by the query node that has sent the data query request, when it is determined based on the feedback information of the query node that the data storage cluster where the query node is located does not store the data to be migrated, the data query request is sent to the query node in the next data storage cluster, and the source data storage cluster is continued to be determined based on the feedback information of the next query node until all the query nodes in the data storage cluster are traversed.

[0095] As an example, see Figure 7 , Figure 7 Schematic diagram of the distributed storage system provided by the embodiment of the present application. The routing node 92 responds to the data query request for the data to be migrated sent by the scheduling node 91, and sends a query to the data storage cluster (for example, Figure 7 The query node in the data storage cluster 93 shown in the figure sends a data query request. After receiving the data query request, the query node in the data storage cluster 93 responds to the data query request and queries whether the data storage cluster 93 stores the data to be migrated. When the data storage cluster 93 does not store the data to be migrated, the query request is sent to the query node in the data storage cluster 94 until all the query nodes in the data storage clusters are traversed.

[0096] In this way, by sending data query requests to query nodes in the data storage clusters in sequence, each data storage cluster in the distributed storage system will not respond to the data query request at the same time, thereby effectively reducing the overall load of the distributed storage system.

[0097] In other embodiments, when sending data query requests to the query nodes in the data storage cluster in sequence, after receiving feedback information returned by the query node that has sent the data query request, when it is determined based on the feedback information of the query node that the data storage cluster where the query node is located does not store the data to be migrated, the data query request is sent to the query node in the next data storage cluster, and the source data storage cluster is continued to be determined based on the feedback information of the next query node until the source data storage cluster in the data storage cluster is determined.

[0098] As an example, see Figure 7 , Figure 7Schematic diagram of the distributed storage system provided by the embodiment of the present application. The routing node 92 responds to the data query request for the data to be migrated sent by the scheduling node 91, and sends a query to the data storage cluster (for example, Figure 7 The query node in the data storage cluster 93 shown in the figure sends a data query request. After receiving the data query request, the query node in the data storage cluster 93 responds to the data query request and queries whether the data storage cluster 93 stores data to be migrated. When the data storage cluster 93 stores data to be migrated, the data storage cluster 93 is determined as the source data storage cluster.

[0099] In this way, by sending data query requests to the query nodes in the data storage cluster in sequence until the source data storage cluster in the data storage cluster is determined, the time for determining the source data storage cluster is effectively saved and the efficiency of determining the source data storage cluster is effectively improved.

[0100] As an example, see Figure 7 , Figure 7 Schematic diagram of the distributed storage system provided by the embodiment of the present application. The routing node 92 responds to the data query request for the data to be migrated sent by the scheduling node 91, and sends a query to the data storage cluster (for example, Figure 7 The query nodes in the data storage cluster 93, data storage cluster 94, and data storage cluster 95 shown in the figure send a data query request. After receiving the data query request, the query node in the data storage cluster 93 responds to the data query request and queries whether the data storage cluster 93 stores the data to be migrated. After receiving the data query request, the query node in the data storage cluster 94 responds to the data query request and queries whether the data storage cluster 93 stores the data to be migrated.

[0101] In some embodiments, by simultaneously sending a data query request to each query node in the data storage cluster, each query node can query the feedback information in parallel, which can effectively improve the query efficiency of the feedback information, and can perform storage information analysis on the entire data storage cluster in the distributed storage system, which can effectively ensure query accuracy and query efficiency. The following will explain the subsequent steps of step 101 based on the implementation method of simultaneously sending a data query request to each query node in the data storage cluster.

[0102] In step 102, feedback information returned by the query node in response to the data query request is received.

[0103] In some embodiments, the feedback information is used to indicate whether the corresponding data storage cluster stores data to be migrated.

[0104] As an example, see Figure 7The routing node 92 receives feedback information returned by the query node in the data storage cluster 93 for the data query request, the routing node 92 receives feedback information returned by the query node in the data storage cluster 94 for the data query request, and the routing node 92 receives feedback information returned by the query node in the data storage cluster 95 for the data query request.

[0105] In step 103, based on the feedback information, the data storage cluster storing the data to be migrated is determined as the source data storage cluster, and the storage node storing the data to be migrated in the source data storage cluster is determined as the target storage node.

[0106] In some embodiments, since the feedback information indicates whether the corresponding data storage cluster stores data to be migrated, when the feedback information indicates that the corresponding data storage cluster stores data to be migrated, the corresponding data storage cluster can be determined as the source data storage cluster; when the feedback information indicates that the corresponding data storage cluster does not store data to be migrated, the corresponding data storage cluster may not be determined as the source data storage cluster.

[0107] As an example, see Figure 7 Based on the feedback information from data storage cluster 93, data storage cluster 94 and data storage cluster 95, routing node 92 determines the source data storage cluster storing the data to be migrated and the target storage node storing the data to be migrated in the source data storage cluster from data storage cluster 93, data storage cluster 94 and data storage cluster 95.

[0108] In some embodiments, see Figure 8 , Figure 8 It is a flowchart of the data migration method provided in an embodiment of the present application. Figure 8 The illustrated step 103 may be implemented by executing the following steps 1031 to 1032 for the data storage cluster, which will be described below.

[0109] In step 1031 , when the feedback information returned by the data storage cluster indicates that the data storage cluster stores data to be migrated, the data storage cluster is determined as a source data storage cluster.

[0110] In some embodiments, the source data storage cluster is a data storage cluster storing data to be migrated. The data to be migrated stored in the source data storage cluster can be migrated to the corresponding target data storage cluster through the data migration method provided in the embodiments of the present application.

[0111] As an example, see Figure 7When the feedback information fed back by the data storage cluster 93 indicates that the data storage cluster 93 stores data to be migrated, the data storage cluster 93 is determined as the source data storage cluster. When the data storage cluster 94 stores data to be migrated, the data storage cluster 94 is determined as the source data storage cluster. When the feedback information fed back by the data storage cluster 94 indicates that the data storage cluster 94 stores data to be migrated, the data storage cluster 94 is determined as the source data storage cluster. There is at least one source data storage cluster among the data storage cluster 93, the data storage cluster 94 and the data storage cluster 95, that is, there is at least one source data storage cluster among the above-mentioned multiple data storage clusters.

[0112] In step 1032, a target storage node storing the data to be migrated is determined from the storage nodes in the source data storage cluster.

[0113] As an example, see Figure 7 When the source data storage cluster is data storage cluster 93, the target storage node storing the data to be migrated is determined from storage nodes 931 and 932 in data storage cluster 93. When the source data storage cluster is data storage cluster 94, the target storage node storing the data to be migrated is determined from storage nodes 941 and 942 in data storage cluster 94. When the source data storage cluster is data storage cluster 95, the target storage node storing the data to be migrated is determined from storage nodes 951 and 952 in data storage cluster 95.

[0114] As an example, see Figure 7 When the source data storage cluster is data storage cluster 93, storage node 932 in data storage cluster 93 stores data to be migrated, and storage node 932 is the target storage node for storing the data to be migrated. When the source data storage cluster is data storage cluster 94, storage node 941 and storage node 942 in data storage cluster 94 both store data to be migrated, and storage node 941 and storage node 942 are the target storage nodes for storing the data to be migrated.

[0115] In some embodiments, the above-mentioned step 1032 can be implemented in the following manner: when the feedback information carries the storage address of the data to be migrated, the storage address is compared with the address range of the storage nodes in the source data storage cluster to obtain a first comparison result of the storage node; based on the first comparison result, the target storage node is determined from the storage nodes in the source data storage cluster; when the feedback information does not carry the storage address of the data to be migrated, a node query request is sent to the source data storage cluster, and the node query request is used to query the target storage node among the storage nodes in the source data storage cluster; and the node identifier of the target storage node storing the data to be migrated returned by the source data storage cluster in response to the node query request.

[0116] In some embodiments, the first comparison result is used to indicate whether the storage address of the data to be migrated matches the address range of the corresponding storage node.

[0117] In some embodiments, since the feedback information is generated at the query node and sent to the routing node, from the perspective of the routing node, the received feedback information may carry the storage address of the data to be migrated, or may not carry the storage address of the data to be migrated. When the feedback information carries the storage address of the data to be migrated, the routing node can directly determine the target storage node based on the storage address of the data to be migrated. When the feedback information does not carry the storage address of the data to be migrated, the routing node can send a node query request to the source data storage cluster, and receive the node identifier of the target storage node storing the data to be migrated returned by the source data storage cluster in response to the node query request, thereby determining the target storage node.

[0118] As an example, see Figure 7 When the source data storage cluster is data storage cluster 93, when the feedback information of data storage cluster 93 carries the storage address of the data to be migrated, the storage address is compared with the address range of the storage nodes (storage node 931 and storage node 932) in the source data storage cluster 93 to obtain a first comparison result of the storage nodes (storage node 931 and storage node 932); based on the first comparison result, the target storage node is determined from the storage nodes (storage node 931 and storage node 932) in the source data storage cluster 93.

[0119] As an example, when the feedback information of the data storage cluster carries the storage address (19.21.21.26) of the data to be migrated, the storage address (19.21.21.26) is compared with the address range (19.21.21.00-19.21.21.26) of the storage node A in the source data storage cluster and the address range (19.21.21.27-19.21.21.36) of the storage node B to obtain a first comparison result of the storage node A (indicating that the storage address of the data to be migrated is within the address range of the storage node A), as well as a first comparison result of the storage node B (indicating that the storage address of the data to be migrated is not within the address range of the storage node B).

[0120] As an example, when the feedback information of the data storage cluster carries the storage address of the data to be migrated (19.21.21.25-19.21.21.26), the storage address (19.21.21.25-19.21.21.26) is compared with the address range (19.21.21.00-19.21.21.26) of the storage node A and the address range (19.21.21.27-19.21.21.36) of the storage node B in the source data storage cluster to obtain a first comparison result of the storage node A (indicating that the storage address of the data to be migrated is within the address range of the storage node A), as well as a first comparison result of the storage node B (indicating that the storage address of the data to be migrated is not within the address range of the storage node B).

[0121] As an example, see Figure 7 When the source data storage cluster is the data storage cluster 94, since the storage address of the data to be migrated is not carried in the feedback information, the data volume of the feedback information can be effectively reduced, so that the transmission speed of the feedback information in the communication process is faster, and the communication efficiency is effectively improved. When the storage address of the data to be migrated is not carried in the feedback information of the data storage cluster 94, the routing node 92 sends a node query request to the source data storage cluster 94, and the node query request is used to query the target storage node in the storage nodes (storage node 941 and storage node 942) in the source data storage cluster 94 (the storage node in the source data storage cluster responds to the node query request, queries the storage information in the corresponding storage node for indicating whether the data to be migrated is stored, and feeds back the storage information to the query node, and the query node determines the node identifier of the target storage node storing the data to be migrated based on the storage information of the storage node, and sends the node identifier to the routing node); the routing node receives the node identifier of the target storage node storing the data to be migrated returned by the source data storage cluster 94 in response to the node query request.

[0122] In some embodiments, the above-mentioned determination of the target storage node from the storage nodes in the source data storage cluster based on the first comparison result can be achieved in the following manner: perform the following processing on the first comparison result of the storage node: when the first comparison result indicates that the storage address is within the address range of the corresponding storage node, determine the corresponding storage node as the target storage node storing the data to be migrated; when the first comparison result indicates that the storage address is not within the address range of the corresponding storage node, determine the corresponding storage node as other storage nodes that do not store the data to be migrated.

[0123] As an example, see Figure 7 When the source data storage cluster is data storage cluster 95, the following processing is performed on storage node 951 and storage node 952 in data storage cluster 95: when the first comparison result indicates that the storage address is within the address range of the corresponding storage node (storage node 951 / storage node 952), the corresponding storage node (storage node 951 / storage node 952) is determined as the target storage node storing the data to be migrated; when the first comparison result indicates that the storage address is not within the address range of the corresponding storage node (storage node 951 / storage node 952), the corresponding storage node (storage node 951 / storage node 952) is determined as other storage nodes that do not store the data to be migrated.

[0124] In this way, when the storage address of the data to be migrated is carried in the feedback information, the target storage node is determined from the storage nodes in the source data storage cluster by comparing the storage address with the address range of the storage nodes in the source data storage cluster; when the storage address of the data to be migrated is not carried in the feedback information, a node query request is sent to the source data storage cluster, and the node identifier of the target storage node storing the data to be migrated returned by the source data storage cluster in response to the node query request. In this way, by determining whether the storage address of the data to be migrated is carried in the feedback information, different methods are used to determine the target storage node in the source data storage cluster, thereby achieving effective determination of the target storage node.

[0125] In some embodiments, after the above step 102, the data to be migrated can also be read in the following manner: in response to a data read request for the data to be migrated sent by the scheduling node, forward the data read request to the query node in the source data storage cluster; receive the data to be migrated read by the query node in response to the data read request; and send the data to be migrated to the scheduling node.

[0126] In some embodiments, the data read request is used to trigger the query node to read the data to be migrated from the target storage node.

[0127] In some embodiments, before data migration, after determining the source data storage cluster storing the data to be migrated and the target storage node in the source data storage cluster storing the data to be migrated, when the scheduling node has a need to read the data to be migrated, it can send a data read request for the data to be migrated to the routing node. After the routing node responds to the data read request for the data to be migrated sent by the scheduling node, the routing node already knows the source data storage cluster storing the data to be migrated, then the routing node can forward the data read request to the query node in the source data storage cluster. After receiving the data read request, the query node in the source data storage cluster reads the data to be migrated and sends the data to be migrated to the routing node. The routing node receives the data to be migrated read by the query node in response to the data read request, and sends the data to be migrated to the scheduling node.

[0128] In some embodiments, the number of source data storage clusters storing data to be migrated may be at least one, then the above-mentioned forwarding of data read requests to query nodes in the source data storage cluster may be implemented in the following manner: when the number of source data storage clusters is multiple, forwarding data read requests to query nodes in the source data storage cluster, or forwarding data read requests to query nodes in any one of the source data storage clusters; when the number of source data storage clusters is one, forwarding data read requests to query nodes in the source data storage cluster.

[0129] In some embodiments, when there are multiple source data storage clusters, since the data to be migrated stored in each source data storage cluster is the same, the data read request can be forwarded to the query node in the source data storage cluster, or the data read request can be forwarded to the query node in any one of the source data storage clusters, thereby achieving effective reading of the data to be migrated.

[0130] In some embodiments, when a data read request is forwarded to a query node in a source data storage cluster, the above-mentioned sending of the data to be migrated to the scheduling node can be achieved in the following manner: when multiple copies of the data to be migrated read by the query nodes in response to the data read request are received, deduplication processing is performed on the multiple copies of the data to be migrated to obtain target data to be migrated; and the target data to be migrated is sent to the scheduling node.

[0131] In this way, before data migration, after determining the source data storage cluster storing the data to be migrated and the target storage node storing the data to be migrated in the source data storage cluster, when the scheduling node has a need to read the data to be migrated, the routing node responds to the data read request for the data to be migrated sent by the scheduling node. At this time, the routing node already knows the source data storage cluster storing the data to be migrated, then the routing node can forward the data read request to the query node in the source data storage cluster. After receiving the data read request, the query node in the source data storage cluster reads the data to be migrated, thereby realizing effective reading of the data to be migrated before data migration.

[0132] In step 104, the node identifier of the target storage node is sent to the scheduling node.

[0133] In some embodiments, the node identifier is used to trigger the scheduling node to migrate the data to be migrated from the source data storage cluster to the target data storage cluster.

[0134] As an example, see Figure 7 When the target storage node is storage node 942, routing node 92 sends the node identifier of storage node 942 to scheduling node 91 to trigger scheduling node 91 to migrate the data to be migrated from the source data storage cluster to the target data storage cluster.

[0135] As an example, see Figure 7 When the target storage node is storage node 951, routing node 92 sends the node identifier of storage node 951 to scheduling node 91 to trigger scheduling node 91 to migrate the data to be migrated from the source data storage cluster to the target data storage cluster.

[0136] In some embodiments, after step 104, see Fig. 9 , Fig. 9 10 is a flow chart of the data migration method provided in an embodiment of the present application. The routing node can also send and receive the data to be migrated through the following steps 105 to 107.

[0137] In step 105 , in response to a data read request for the data to be migrated sent by the scheduling node, the data read request is forwarded to a query node in the source data storage cluster.

[0138] In some embodiments, the data read request is used to trigger the query node to read the data to be migrated from the target storage node.

[0139] As an example, when the source data storage cluster is data storage cluster 93 , routing node 92 forwards the data read request for the data to be migrated sent by scheduling node 91 to the query node in data storage cluster 93 .

[0140] In some embodiments, when the source data storage cluster is data storage cluster 94 , routing node 92 forwards the data read request for the data to be migrated sent by scheduling node 91 to the query node in data storage cluster 94 .

[0141] In step 106, the data to be migrated read by the query node in response to the data read request is received.

[0142] As an example, see Figure 7 The routing node 92 receives the data to be migrated read by the query node in response to the data read request.

[0143] In step 107, the received data to be migrated is sent to the target data storage cluster.

[0144] As an example, see Figure 7 , when the target data storage cluster is the data storage cluster 95 , the received data to be migrated is sent to the data storage cluster 95 .

[0145] In some embodiments, before the above step 107, the reference storage node in the target data storage cluster can also be determined in the following manner: sending a storage space query request to the query node in the target data storage cluster; receiving the remaining storage space of the storage node returned by the query node in response to the storage space query request, and combining the remaining storage space and the amount of data to be migrated, determining the reference storage node from the storage nodes of the target data storage cluster; and sending the target node identifier of the reference storage node to the query node in the target data storage cluster.

[0146] In some embodiments, the storage space query request is used to trigger the query node to query the remaining storage space of the storage nodes in the target data storage cluster.

[0147] In some embodiments, the target node identifier is used to trigger the query node to send the data to be migrated to the reference storage node after receiving the data to be migrated.

[0148] As an example, when the target data storage cluster is data storage cluster 95, routing node 92 sends a storage space query request to the query node in data storage cluster 95. The query node in data storage cluster 95 queries the remaining storage space of storage node 951 and storage node 952 in response to the received storage space query request, and sends the queried remaining storage space to routing node 92. Routing node 92 receives the remaining storage space of the storage nodes (storage node 951 and storage node 952) returned by the query node in response to the storage space query request, and determines the reference storage node from the storage nodes (storage node 951 and storage node 952) of the target data storage cluster in combination with the remaining storage space and the amount of data to be migrated. Routing node 92 sends the target node identifier of the reference storage node to the query node in the target data storage cluster.

[0149] In some embodiments, step 107 may be implemented as follows: sending the received data to be migrated to a query node in the target data storage cluster so that the query node creates an index entry for the data to be migrated and sends the data to be migrated to a reference storage node.

[0150] As an example, see Figure 7 When the target data storage cluster is data storage cluster 95, routing node 92 sends the received data to be migrated to the query node in data storage cluster 95, so that the query node in data storage cluster 95 creates an index entry for the data to be migrated and sends the data to be migrated to the reference storage node (for example, storage node 951).

[0151] In some embodiments, the above-mentioned combination of the remaining storage space and the amount of data to be migrated, determining the reference storage node from the storage nodes of the target data storage cluster, can be achieved in the following manner: comparing the remaining storage space with the amount of data to obtain a second comparison result; when the second comparison result indicates that the remaining storage space can accommodate the amount of data to be migrated, determining the storage node as a candidate storage node; when the number of candidate storage nodes is one, determining the candidate storage node as a reference storage node; when the number of candidate storage nodes is multiple, determining the candidate storage node with the smallest remaining storage space as the reference storage node.

[0152] In some embodiments, the data volume is used to indicate the size of the storage space required for the data to be migrated. The second comparison result is used to indicate whether the remaining storage space can accommodate the data volume of the data to be migrated.

[0153] As an example, see Figure 7, for storage node 952, the remaining storage space of storage node 952 is compared with the data volume to obtain a second comparison result of storage node 952. When the second comparison result indicates that the remaining storage space can accommodate the data volume to be migrated, storage node 952 is determined as a candidate storage node; for storage node 951, the remaining storage space of storage node 951 is compared with the data volume to obtain a second comparison result of storage node 951. When the second comparison result indicates that the remaining storage space can accommodate the data volume to be migrated, storage node 951 is determined as a candidate storage node. When there are multiple candidate storage nodes, the candidate storage node with the smallest remaining storage space is determined as a reference storage node (for example, if the remaining storage space of storage node 951 is greater than the remaining storage space of storage node 952, storage node 952 is determined as a reference storage node).

[0154] In this way, by combining the remaining storage space of the storage node and the amount of data to be migrated, a reference storage node is determined from the storage nodes of the target data storage cluster, thereby achieving optimal adaptation of the remaining storage space of the determined reference storage node and the amount of data to be migrated, effectively improving the efficient use of the storage space of the storage nodes in the target data storage cluster and effectively improving the utilization rate of the storage space.

[0155] In some embodiments, after the above step 106, data deletion can be performed in the following manner: sending a data deletion request to a query node in the source data storage cluster; and sending a data deletion request to a target storage node in the source data storage cluster so that the target storage node deletes the data to be migrated stored in the target storage node; and receiving indication information returned by the query node in the source data storage cluster in response to the data deletion request.

[0156] In some embodiments, the data deletion request is used to trigger the query node to delete the index entry of the data to be migrated stored in the query node.

[0157] In some embodiments, the indication information is used to trigger the routing node to send the received data to be migrated to the target data storage cluster.

[0158] As an example, see Figure 7 When the source data storage cluster is the data storage cluster 93, the routing node 92 sends a data deletion request to the query node in the data storage cluster 93, and sends a data deletion request to the target storage node (for example, storage node 931) in the source data storage cluster, so that the target storage node deletes the data to be migrated stored in the target storage node, and receives the indication information returned by the query node in the data storage cluster 93 in response to the data deletion request.

[0159] In some embodiments, the above step 107 may be implemented in the following manner: in response to the received indication information, the data to be migrated is sent to the target data storage cluster.

[0160] As an example, see Figure 7 , the routing node 92 sends the data to be migrated to the target data storage cluster in response to the received instruction information.

[0161] In this way, after receiving the data to be migrated read by the query node in response to the data reading request, the data to be migrated stored in the source data storage cluster is deleted, and the routing node then migrates the data to be migrated to the target data storage cluster. This enables the storage space of the source data storage cluster to be quickly released during the data migration process, effectively reducing the data storage pressure of the source data storage cluster.

[0162] In some embodiments, after the routing node sends the data to be migrated to the target data storage cluster in response to the received indication information, the routing node can also implement reading of the data to be migrated in the following manner: in response to a data read request for the data to be migrated sent by the scheduling node, forward the data read request to the query node in the target data storage cluster; receive the data to be migrated read by the query node in response to the data read request; and send the data to be migrated to the scheduling node.

[0163] In some embodiments, the data read request is used to trigger the query node to read the data to be migrated from the target data storage cluster.

[0164] In some embodiments, after the data to be migrated is sent to the target data storage cluster and the data to be migrated in the source data storage cluster has been deleted, a data read request for the data to be migrated is received from the scheduling node. In response to the data read request for the data to be migrated sent by the scheduling node, the routing node knows that only the target data storage cluster stores the data to be migrated. Then, the routing node can forward the data read request to the query node in the target data storage cluster, so that the query node in the target data storage cluster queries the stored data to be migrated and sends the queried data to be migrated to the routing node. After receiving the data to be migrated read by the query node in response to the data read request, the routing node sends the data to be migrated to the scheduling node, so that the scheduling node can realize data query when only the data to be migrated is stored in the target data storage cluster during the data migration process.

[0165] In this way, after receiving the data to be migrated read by the query node in response to the data reading request, the data to be migrated stored in the source data storage cluster is deleted, and the routing node then migrates the data to be migrated to the target data storage cluster. In this way, in the process of deleting the data to be migrated stored in the source data storage cluster, the data to be migrated always remains in the routing node. When the routing node receives a data reading request for the data to be migrated, it does not need to obtain the data to be migrated from the source data storage cluster and the target data storage cluster, but can directly read the data to be migrated from the local routing node, thereby effectively improving the efficiency of obtaining the data to be migrated.

[0166] In some other embodiments, the above step 107 may also be implemented in the following manner: in response to receiving the data to be migrated, sending the data to be migrated to the target data storage cluster.

[0167] As an example, see Figure 7 In response to receiving the data to be migrated, the routing node 92 sends the data to be migrated to the target data storage cluster. In other embodiments, after the above step 107, data deletion can also be performed in the following manner: sending a data deletion request to a query node in the source data storage cluster, and sending a data deletion request to a target storage node in the source data storage cluster, so that the target storage node deletes the data to be migrated stored in the target storage node; receiving indication information returned by the query node in the source data storage cluster in response to the data deletion request.

[0168] In some embodiments, the data deletion request is used to trigger the query node to delete the index entry of the data to be migrated stored in the query node.

[0169] In some embodiments, the indication information is used to indicate that there is no data to be migrated in the source data storage cluster.

[0170] As an example, see Figure 7 When the source data storage cluster is the data storage cluster 93, the routing node 92 sends a data deletion request to the query node in the data storage cluster 93, and sends a data deletion request to the target storage node (for example, storage node 931) in the source data storage cluster, so that the target storage node deletes the data to be migrated stored in the target storage node, and receives the indication information returned by the query node in the data storage cluster 93 in response to the data deletion request.

[0171] In some embodiments, after the data to be migrated is sent to the target data storage cluster and before a data deletion request is sent to the query node in the source data storage cluster, at this time, the data to be migrated is stored in both the source data storage cluster and the target data storage cluster. At this time, the scheduling node can implement the reading of the data to be migrated in the following manner: in response to the data reading request for the data to be migrated sent by the scheduling node, forward the data reading request to the source data storage cluster or the target data storage cluster; receive the data to be migrated read by the source data storage cluster or the target data storage cluster; and send the data to be migrated to the scheduling node.

[0172] In some embodiments, during the process of data migration, when the data to be migrated is stored in both the source data storage cluster and the target data storage cluster, the reading of the data to be migrated can be achieved at this time. The data reading request can be forwarded to any one of the source data storage cluster and the target data storage cluster, and the data to be migrated read by the source data storage cluster or the target data storage cluster can be received, thereby achieving effective reading of the data to be migrated.

[0173] In some embodiments, after the data to be migrated is sent to the target data storage cluster and before the indication information returned by the query node in the source data storage cluster in response to the data deletion request is received, the data to be migrated may still be stored in the source data storage cluster, or may not be stored in the source data storage cluster. At the same time, the data to be migrated is stored in the target data storage cluster. At this time, the scheduling node can realize the reading of the data to be migrated in the following manner: in response to the data read request for the data to be migrated sent by the scheduling node, forward the data read request to the source data storage cluster and the target data storage cluster; receive the data to be migrated returned in response to the data read request, and when the data to be migrated returned by the source data storage cluster and the data to be migrated returned by the target data storage cluster are received, compare the data to be migrated returned by the source data storage cluster and the target data storage cluster to obtain a comparison result; when the comparison result indicates that the data to be migrated returned by the source data storage cluster and the target data storage cluster are the same, deduplicate the returned data to be migrated to obtain the target data to be migrated; and send the target data to be migrated to the scheduling node.

[0174] In some embodiments, when the data to be migrated may still be stored in the source data storage cluster, or may not be stored in the source data storage cluster, and the data to be migrated is stored in the target data storage cluster, a data read request can be forwarded to the source data storage cluster and the target data storage cluster; the data to be migrated returned in response to the data read request is received, and when the data to be migrated returned by the source data storage cluster and the data to be migrated returned by the target data storage cluster are received, the data to be migrated returned by the source data storage cluster and the target data storage cluster are compared to obtain a comparison result; when the comparison result indicates that the data to be migrated returned by the source data storage cluster and the target data storage cluster are the same, the returned data to be migrated are deduplicated to obtain the target data to be migrated; and the target data to be migrated is sent to the scheduling node.

[0175] In this way, after receiving the data to be migrated, the data to be migrated is sent to the target data storage cluster, and then the data to be migrated in the source data storage cluster is deleted. Therefore, before the data to be migrated is stored in the target data storage cluster, the data to be migrated is still backed up in the source data storage cluster. When data loss occurs during the migration process, data can be traced from the source data storage cluster in time, thereby effectively improving the security of data migration.

[0176] In some embodiments, see Fig.10 , Fig.10 is a flow chart of the data migration method provided in the embodiment of the present application, which can be Fig.10 Steps 201 to 203 shown determine a target data storage cluster.

[0177] In step 201, a storage space query request is sent to a query node in a data storage cluster.

[0178] In some embodiments, the storage space query request is used to trigger the query node to query the overall remaining storage space of the storage nodes in the corresponding target data storage cluster.

[0179] As an example, see Figure 7 , routing node 92 sends a storage space query request to the query nodes in data storage cluster 93 , data storage cluster 94 , and data storage cluster 95 .

[0180] In some embodiments, the query node queries the remaining storage space of the storage nodes in the corresponding data storage cluster in response to the received storage space query request, sums the remaining storage space of the storage nodes to obtain the overall remaining storage space, and sends the overall remaining storage space to the routing node.

[0181] In other embodiments, the query node queries the remaining storage space of the storage nodes in the corresponding data storage cluster in response to the received storage space query request, and sends the remaining storage space of the storage nodes to the routing node. The routing node sums the remaining storage space of the storage nodes to obtain the overall remaining storage space.

[0182] In step 202, the overall remaining storage space returned by the data storage cluster in response to a storage space query request is received.

[0183] As an example, see Figure 7 , routing node 92 receives the overall remaining storage space returned by data storage cluster 93 for storage space query request. Routing node 92 receives the overall remaining storage space returned by data storage cluster 94 for storage space query request. Routing node 92 receives the overall remaining storage space returned by data storage cluster 95 for storage space query request.

[0184] In step 203, based on the overall remaining storage space, a target data storage cluster is determined from a plurality of data storage clusters.

[0185] In some embodiments, the target data storage cluster and the source data storage cluster are different data storage clusters.

[0186] As an example, see Figure 7 The routing node 92 determines a target data storage cluster from multiple data storage clusters based on the overall remaining storage space.

[0187] In some embodiments, see Fig.11 , Fig.11 is a flow chart of a data migration method provided in an embodiment of the present application, Fig.10 The illustrated step 203 may be performed by Fig.11 Steps 2031 to 2033 are implemented as shown.

[0188] In step 2031, the overall remaining storage space is compared with the amount of data to be migrated to obtain a third comparison result; when the third comparison result indicates that the overall remaining storage space can accommodate the amount of data to be migrated, the data storage cluster is determined as a candidate data storage cluster.

[0189] As an example, see Figure 7 , with respect to the overall remaining storage space of the data storage cluster 93, the overall remaining storage space of the data storage cluster 93 is compared with the amount of data to be migrated to obtain a third comparison result. When the third comparison result indicates that the overall remaining storage space can accommodate the amount of data to be migrated, the data storage cluster 93 is determined as a candidate data storage cluster.

[0190] As an example, see Figure 7 , with respect to the overall remaining storage space of the data storage cluster 94, the overall remaining storage space of the data storage cluster 94 is compared with the amount of data to be migrated to obtain a third comparison result. When the third comparison result indicates that the overall remaining storage space can accommodate the amount of data to be migrated, the data storage cluster 94 is determined as a candidate data storage cluster.

[0191] As an example, see Figure 7 , with respect to the overall remaining storage space of the data storage cluster 95, the overall remaining storage space of the data storage cluster 95 is compared with the amount of data to be migrated to obtain a third comparison result. When the third comparison result indicates that the overall remaining storage space cannot accommodate the amount of data to be migrated, the data storage cluster 95 is determined as a non-candidate data storage cluster.

[0192] In step 2032 , when the number of candidate data storage clusters is one, the candidate data storage cluster is determined as a target data storage cluster.

[0193] As an example, see Figure 7 When the candidate data storage cluster is the data storage cluster 93, the number of the candidate data storage clusters is one, and the data storage cluster 93 is determined as the target data storage cluster.

[0194] In step 2033, when there are multiple candidate data storage clusters, the candidate data storage cluster with the smallest overall remaining storage space is determined as the target data storage cluster.

[0195] As an example, see Figure 7 When the candidate data storage clusters are data storage cluster 93 and data storage cluster 94, the number of candidate data storage clusters is multiple, and the overall remaining storage space of data storage cluster 93 is greater than the overall remaining storage space of data storage cluster 94, then data storage cluster 94 is determined as the target data storage cluster.

[0196] See also Fig.12 , Fig.12 is a flow chart of the data migration method provided in the embodiment of the present application, which will be combined with Fig.12 Steps 301 to 302 are shown for explanation. The data migration method provided in the embodiment of the present application can be implemented by a server or a terminal alone, or by a server and a terminal in collaboration. The following will take the implementation of the server alone as an example to explain the data migration method provided in the embodiment of the present application from the perspective of the scheduling node.

[0197] In step 301, a data query request is sent to a routing node, so that the routing node responds to the data query request, sends the data query request to the query node in the data storage cluster, and receives feedback information returned by the query node for the data query request, and based on the feedback information, determines that the data storage cluster storing the data to be migrated is the source data storage cluster, and determines that the storage node storing the data to be migrated in the source data storage cluster is the target storage node.

[0198] In some embodiments, the data query request is used to query whether the data to be migrated is stored in the data storage cluster, the data storage cluster includes a query node and a storage node, and the data storage cluster includes a target data storage cluster.

[0199] In step 302, in response to the node identifier of the target storage node sent by the routing node, the data to be migrated is migrated from the source data storage cluster to the target data storage cluster.

[0200] In some embodiments, the above step 302 can be implemented in the following manner: in response to the node identifier of the target storage node sent by the routing node, a data migration request for the data to be migrated is sent to the routing node, so that the routing node forwards the data migration request to the query node in the source data storage cluster, receives the data to be migrated read by the query node in response to the data migration request, and sends the received data to be migrated to the target data storage cluster.

[0201] In some embodiments, the data migration request is used to trigger the query node to read the data to be migrated from the target storage node.

[0202] In some embodiments, the scheduling node can also read the data to be migrated in the following manner: send a data acquisition request for the data to be migrated to the routing node, so that the routing node forwards the data reading request to the query node in the target data storage cluster, and receives the data to be migrated sent by the query node, the data reading request is used to trigger the query node to read the data to be migrated; receive the data to be migrated sent by the routing node.

[0203] See also Fig.13 , Fig.13 is a flow chart of the data migration method provided in the embodiment of the present application, which will be combined with Fig.13 Steps 401 to 405 are shown for illustration. The data migration method provided in the embodiment of the present application can be implemented by a server or a terminal alone, or by a server and a terminal in collaboration. The following takes the implementation of a server alone as an example to illustrate the data migration method provided in the embodiment of the present application from the perspective of interaction between a scheduling node and a routing node in a distributed storage system.

[0204] In step 401, the scheduling node sends a data query request to the routing node, where the data query request is used to query whether there is data to be migrated stored in the data storage cluster.

[0205] In step 402, the routing node sends a data query request to a query node in a data storage cluster in response to the data query request, and receives feedback information returned by the query node for the data query request. The data storage cluster includes query nodes and storage nodes.

[0206] In step 403, the routing node determines, based on the feedback information, a data storage cluster storing the data to be migrated as a source data storage cluster, and determines a storage node storing the data to be migrated in the source data storage cluster as a target storage node.

[0207] In step 404, the routing node sends the node identifier of the target storage node to the scheduling node.

[0208] In step 405, the scheduling node receives the node identifier of the target storage node, and based on the node identifier, migrates the data to be migrated from the source data storage cluster to the target data storage cluster.

[0209] In this way, the routing node responds to the data query request for the data to be migrated sent by the scheduling node, sends a data query request to the query node in the data storage cluster, and receives feedback information returned by the query node for the data query request. Based on the feedback information, the source data storage cluster storing the data to be migrated and the target storage node storing the data to be migrated in the source data storage cluster are determined from multiple data storage clusters, and the node identifier of the target storage node is sent to the scheduling node, so that the scheduling node migrates the data to be migrated from the source data storage cluster to the target data storage cluster. In this way, the source data storage cluster storing the data to be migrated and the target storage node storing the data to be migrated in the source data storage cluster are determined through the routing node, and the node identifier of the target storage node is sent to the scheduling node through the routing node to migrate the data to be migrated from the source data storage cluster to the target data storage cluster, thereby achieving that during the migration process of the data to be migrated, the distributed storage system always stores the data to be migrated, so that the scheduling node can normally respond to the data reading of the data to be migrated, and the data to be migrated can be read from the distributed storage system, so that the data migration process will not interfere with the data reading service, so that the data reading service can operate normally, and data migration is achieved without the data reading service being aware of it, thereby effectively reducing the impact of data migration on the data reading service.

[0210] The following describes an exemplary application of the embodiments of the present application in an actual application scenario of a distributed storage system.

[0211] The data migration method provided in the embodiments of the present application can be applied to any scenario that requires data migration, and the data migration method provided in the embodiments of the present application can be applied to products in these scenarios, for example, game applications that provide services through distributed storage systems, commodity classification applications that provide services through distributed storage systems, and other application products that require services through distributed storage systems. Fig.12 Fig.14 , Fig.12 Fig.14 It is a principle schematic diagram of the application scenario of the data migration method provided in the embodiment of the present application. The APP product in the terminal 82 responds to the operation of the user 81 on the APP product, generates a data acquisition request adapted to the operation, and sends the data acquisition request to the distributed storage system 83. When the number of data acquisition requests for the data to be migrated received by the distributed storage system is too large, the response pressure of the storage node storing the data to be migrated will increase sharply. At this time, the data migration method provided in the embodiment of the present application can be used to realize the data migration of the data to be migrated, so as to reduce the response pressure of the distributed storage system.

[0212] As an example, see Fig.12 Fig.14 In a game application provided by a distributed storage system, the game APP product in the terminal 82 generates a data acquisition request adapted to the operation in response to the operation of the game player 81 on the APP product, and sends the data acquisition request to the distributed storage system 83. When the number of data acquisition requests for the data to be migrated received by the distributed storage system is too large, the response pressure of the storage node storing the data to be migrated will increase sharply. At this time, the data migration method provided in the embodiment of the present application can be used to realize the data migration of the data to be migrated, so as to reduce the response pressure of the distributed storage system.

[0213] As an example, see Fig.12 Fig.14 In a commodity classification application that provides services through a distributed storage system, the commodity classification APP product in the terminal 82 responds to the operation of the game player 81 on the APP product, generates a data acquisition request adapted to the operation, and sends the data acquisition request to the distributed storage system 83. When the number of data acquisition requests for the data to be migrated received by the distributed storage system is too large, the response pressure of the storage node storing the data to be migrated will increase sharply. At this time, the data migration method provided in the embodiment of the present application can be used to realize the data migration of the data to be migrated, so as to reduce the response pressure of the distributed storage system.

[0214] In some embodiments, see Figure 7, add a routing strategy for the path of the table to be migrated on the routing node: the routing strategy uses the custom implemented WRITER_FIRST strategy, which implements routing the write operation to the first configured HDFS cluster for execution, and the read operation is to read from all HDFS clusters. During configuration, the embodiment of the present application configures "HDFS Cluster 93" to the first position, which ensures that all new data will be written to the target "HDFS Cluster 93". This configuration can avoid inconsistencies in data migration caused by newly written data being written to "HDFS Cluster 92" during data migration. When reading data, the client will read data from both "HDFS Cluster 92" and "HDFS Cluster 93" HDFS clusters at the same time to ensure that both new and old data can be read at the same time.

[0215] In some embodiments, the number of HDFS clusters can be multiple. The process of data query is described by taking two HDFS clusters as the number. In the above data migration process, or before data migration, or after data migration is completed, the data to be migrated can be queried. Specifically, the scheduling node queries the routing node for the files under the path / user / hive / warehouse / test_db.db / test_table. The routing node routes the query request to the Namenode in the cluster storing the data to be migrated in the two HDFS clusters for execution, and returns the file list and Datanode to the application.

[0216] The following describes the data migration process in combination with steps 1 to 6, taking "HDFS Cluster 1" as the source data storage cluster and "HDFS Cluster 2" as the target data storage cluster as an example. Step 1: The scheduling node submits a file creation request under the path / user / hive / warehouse / test_db.db / test_table to the routing node. Step 2: The routing node routes the write request to the Namenode in "HDFS Cluster 2" for execution, and returns the Datanode information for storing the newly created file. Step 3: The scheduling node selects the Datanode that stores the file data according to the data returned in step 2. Step 4: Migrate the data to the Datanode returned in step 3. Step 5: Modify the routing policy of the scheduling node and configure the path / user / hive / warehouse / test_db.db / test_table only to "HDFS Cluster 2". The configuration command is as follows: $HADOOP_HOME / bin / hdfs dfs routing node admin-add / user / hive / warehouse / test_db.db / test_table HDFS Cluster 2 / user / hive / wareho use / test_db.db / test_table. Step 6: Delete the directory / user / hive / warehouse / test_db.db / test_table in HDFS Cluster 1.

[0217] During the data migration process, if the business party initiates a request to read the data in the table, the custom-implemented routing strategy will route the path query request to "HDFS Cluster 1" and / or "HDFS Cluster 2", ensuring that the client can query the files that have not been migrated, have been migrated, and have been newly written.

[0218] In this way, new data can be written during the data migration process to avoid writing data during migration, which may cause inconsistency between the migrated data and the original data. During data migration, the business side can still read the files that have not been migrated, have been migrated, and have been newly written to ensure business continuity.

[0219] In some embodiments, before the scheduling node deletes the files in "HDFS Cluster 1", the migrated files will exist in the two HDFS clusters "HDFS Cluster 1" and "HDFS Cluster 2". When querying the two HDFS clusters "HDFS Cluster 1" and "HDFS Cluster 2" for the data to be migrated, in order to ensure that the client does not read the same file repeatedly, after the client receives the file list information returned by the routing node, it will check whether there are files with the same path, size, and modification time in the returned files. If such files exist, CRC calculation will be performed on the two files. If the calculated CRCs are the same, it indicates that the two files are the same. Therefore, the file returned in "HDFS Cluster 1" will be deleted from the file list returned by the routing node to avoid repeated reading.

[0220] In this way, after data migration, the applications accessing the data will not be aware of it, which can greatly reduce the cost of application transformation and the risks that may arise during the migration process. Through the routing policy configured on the Router, the Router can route the application's requests to the HDFS cluster where the data is stored. After data migration, there is no need to modify the URI metadata in the Hive table. With the routing function implemented by the Router, the application no longer needs to access Hive to obtain the URI, so this information does not need to be modified.

[0221] In this way, the routing node responds to the data query request for the data to be migrated sent by the scheduling node, sends the data query request to the query node in the data storage cluster, and receives the feedback information returned by the query node for the data query request. Based on the feedback information, the source data storage cluster storing the data to be migrated and the target storage node storing the data to be migrated in the source data storage cluster are determined from multiple data storage clusters, and the node identifier of the target storage node is sent to the scheduling node, so that the scheduling node migrates the data to be migrated from the source data storage cluster to the target data storage cluster. In this way, the source data storage cluster storing the data to be migrated and the target storage node storing the data to be migrated in the source data storage cluster are determined by the routing node, and the node identifier of the target storage node is sent to the scheduling node by the routing node to migrate the data to be migrated from the source data storage cluster to the target data storage cluster, so that the data service can operate normally during the migration process of the data to be migrated, and the data migration is realized without the data service being aware of it, so that the impact of data migration on the data service can be effectively reduced.

[0222] It is understandable that in the embodiments of the present application, the data to be migrated and other related data involved, when the embodiments of the present application are applied to specific products or technologies, need to obtain user permission or consent, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0223] The following further describes an exemplary structure of the data migration device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the data migration device 455 of the memory 450 may include: a request sending module, which is used to send the data query request to the query node in the data storage cluster in response to the data query request sent by the scheduling node, and the data query request is used to query whether the data to be migrated is stored in the data storage cluster, the data storage cluster includes a query node and a storage node, and the multiple data storage clusters include a target data storage cluster; a receiving module, which is used to receive feedback information returned by each of the query nodes in response to the data query request, wherein the feedback information is used to indicate whether the data to be migrated is stored in the corresponding data storage cluster; a determination module, which is used to determine, based on the feedback information, that the data storage cluster storing the data to be migrated is the source data storage cluster, and to determine that the storage node storing the data to be migrated in the source data storage cluster is the target storage node; an identification sending module, which is used to send the node identification of the target storage node to the scheduling node, so that the scheduling node migrates the data to be migrated from the source data storage cluster to the target data storage cluster based on the node identification.

[0224] In the above scheme, the above determination module is also used to compare the storage address with the address range of the storage nodes in the source data storage cluster to obtain a first comparison result when the feedback information carries the storage address of the data to be migrated; based on the first comparison result, determine the target storage node from the storage nodes in the source data storage cluster; when the feedback information does not carry the storage address of the data to be migrated, send a node query request to the source data storage cluster, and the node query request is used to query the target storage node among the storage nodes in the source data storage cluster; receive the node identifier of the target storage node storing the data to be migrated returned by the source data storage cluster in response to the node query request.

[0225] In the above scheme, the above determination module is further used to determine the corresponding storage node as the target storage node storing the data to be migrated when the first comparison result indicates that the storage address is within the address range of the corresponding storage node.

[0226] In the above scheme, the above data migration device also includes: a reading module, which is used to forward the data reading request for the data to be migrated sent by the scheduling node to the query node in the source data storage cluster, and the data reading request is used to trigger the query node to read the data to be migrated from the target storage node; a data receiving module is used to receive the data to be migrated read by the query node in response to the data reading request, and send the received data to be migrated to the target data storage cluster.

[0227] In the above scheme, the above data migration device also includes: a target node module, which is used to send a storage space query request to the query node in the target data storage cluster, and the storage space query request is used to trigger the query node to query the remaining storage space of each storage node in the target data storage cluster; receive the remaining storage space of the storage node returned by the query node in response to the storage space query request, and determine the reference storage node from the storage nodes of the target data storage cluster in combination with the remaining storage space and the data volume of the data to be migrated; send the target node identifier of the reference storage node to the query node in the target data storage cluster, and the target node identifier is used to trigger the query node to send the data to be migrated to the reference storage node after receiving the data to be migrated; the above target node module is also used to send the received data to be migrated to the query node in the target data storage cluster, so that the query node creates an index entry for the data to be migrated and sends the data to be migrated to the reference storage node.

[0228] In the above scheme, the above target node module is also used to compare the remaining storage space with the data volume to obtain a second comparison result; when the second comparison result indicates that the remaining storage space can accommodate the data volume to be migrated, the storage node is determined as a candidate storage node; when the number of the candidate storage nodes is one, the candidate storage node is determined as the reference storage node; when the number of the candidate storage nodes is multiple, the candidate storage node with the smallest remaining storage space is determined as the reference storage node.

[0229] In the above scheme, the above data migration device also includes: a deletion module, which is used to send a data deletion request to the query node in the source data storage cluster; the data deletion request is used to trigger the query node to delete the index entry of the data to be migrated stored in the query node, and send a data deletion request to the target storage node in the source data storage cluster, so that the target storage node deletes the data to be migrated stored in the target storage node; receiving the indication information returned by the query node in the source data storage cluster in response to the data deletion request, the indication information is used to trigger the routing node to send the received data to be migrated to the target data storage cluster; the above target node module is also used to send the data to be migrated to the target data storage cluster in response to the received indication information.

[0230] In the above scheme, the above-mentioned target node module is also used to send the data to be migrated to the target data storage cluster in response to receiving the data to be migrated; the above-mentioned deletion module is also used to send a data deletion request to the query node in the source data storage cluster; wherein the data deletion request is used to trigger the query node to delete the index entry of the data to be migrated stored in the query node, and send a data deletion request to the target storage node in the source data storage cluster, so that the target storage node deletes the data to be migrated stored in the target storage node; receive the indication information returned by the query node in the source data storage cluster in response to the data deletion request, and the indication information is used to indicate that the data to be migrated no longer exists in the source data storage cluster.

[0231] In the above scheme, the above data migration device also includes: a query module, used to send a storage space query request to the query node in the data storage cluster, and the storage space query request is used to trigger the query node to query the overall remaining storage space of the storage nodes in the target data storage cluster; receive the overall remaining storage space returned by the data storage cluster in response to the storage space query request; based on the overall remaining storage space, determine the target data storage cluster from the multiple data storage clusters.

[0232] In the above scheme, the above query module is also used to perform the following processing on the overall remaining storage space of the data storage cluster: compare the overall remaining storage space with the data volume of the data to be migrated to obtain a third comparison result; when the third comparison result indicates that the overall remaining storage space can accommodate the data volume of the data to be migrated, determine the data storage cluster as a candidate data storage cluster; when the number of the candidate data storage clusters is one, determine the candidate data storage cluster as the target data storage cluster; when the number of the candidate data storage clusters is multiple, determine the candidate data storage cluster with the smallest overall remaining storage space as the target data storage cluster.

[0233] In the above scheme, the above data migration device also includes: a reading module, which is used to respond to the data reading request for the data to be migrated sent by the scheduling node, forward the data reading request to the query node in the source data storage cluster, and the data reading request is used to trigger the query node to read the data to be migrated from the target storage node; receive the data to be migrated read by the query node in response to the data reading request; and send the data to be migrated to the scheduling node.

[0234] In the above scheme, the reading module is also used to forward the data reading request to the query node in the source data storage cluster when there are multiple source data storage clusters, or to forward the data reading request to any query node in the source data storage cluster; when there is one source data storage cluster, forward the data reading request to the query node in the source data storage cluster; when receiving the data to be migrated read by multiple query nodes in response to the data reading request, deduplicate the multiple copies of the data to be migrated to obtain the target data to be migrated; and send the target data to be migrated to the scheduling node.

[0235] In the above solution, the reading module is further used to send the received data to be migrated to the scheduling node in response to a data reading request for the data to be migrated sent by the scheduling node.

[0236] In the above scheme, the reading module is also used to respond to the data reading request for the data to be migrated sent by the scheduling node, and forward the data reading request to the query node in the target data storage cluster, wherein the data reading request is used to trigger the query node to read the data to be migrated from the target data storage cluster; receive the data to be migrated read by the query node in response to the data reading request; and send the data to be migrated to the scheduling node.

[0237] In the above scheme, the reading module is also used to respond to the data reading request for the data to be migrated sent by the scheduling node, forward the data reading request to the source data storage cluster or the target data storage cluster; receive the data to be migrated read by the source data storage cluster or the target data storage cluster; and send the data to be migrated to the scheduling node.

[0238] In the above scheme, the reading module is also used to respond to the data reading request for the data to be migrated sent by the scheduling node, forward the data reading request to the source data storage cluster and the target data storage cluster; receive the data to be migrated returned in response to the data reading request, and when the data to be migrated returned by the source data storage cluster and the data to be migrated returned by the target data storage cluster are received, compare the data to be migrated returned by the source data storage cluster and the target data storage cluster to obtain a comparison result; when the comparison result indicates that the data to be migrated returned by the source data storage cluster and the target data storage cluster are the same, deduplicate the returned data to be migrated to obtain the target data to be migrated; and send the target data to be migrated to the scheduling node.

[0239] An embodiment of the present application provides a data migration device, which is applied to a scheduling node of a distributed storage system, wherein the distributed storage system includes: the scheduling node, a routing node and a data storage cluster, wherein the data storage cluster includes a source data storage cluster and a target data storage cluster, and the device includes: a request sending module, which is used to send a data query request to the routing node, so that the routing node sends the data query request to the query node in the data storage cluster in response to the data query request, and receives feedback information returned by the query node for the data query request, and based on the feedback information, determines that the data storage cluster storing the data to be migrated is the source data storage cluster, and determines that the storage node storing the data to be migrated in the source data storage cluster is the target storage node; wherein the data query request is used to query whether the data storage cluster stores the data to be migrated, and the data storage cluster includes the query node and a storage node; and a response module, which is used to migrate the data to be migrated from the source data storage cluster to the target data storage cluster in response to the node identifier of the target storage node sent by the routing node.

[0240] In the above scheme, the above response module is also used to send a data migration request for the data to be migrated to the routing node in response to the node identifier of the target storage node sent by the routing node, so that the routing node forwards the data migration request to the query node in the source data storage cluster, and receives the data to be migrated read by the query node for the data migration request, and sends the received data to be migrated to the target data storage cluster; wherein, the data migration request is used to trigger the query node to read the data to be migrated from the target storage node.

[0241] The embodiment of the present application provides a data migration device, which is applied to a distributed storage system. The distributed storage system includes: a scheduling node, a routing node and a data storage cluster, wherein the data storage cluster includes a source data storage cluster and a target data storage cluster. The device includes: a scheduling sending module, which is used for the scheduling node to send a data query request to the routing node, wherein the data query request is used to query whether the data storage cluster stores data to be migrated; a routing response module, which is used for the routing node to send the data query request to the query node in the data storage cluster in response to the data query request, and receive feedback information returned by the query node for the data query request, wherein the data storage cluster includes the query node and a storage node; a routing determination module, which is used for the routing node to determine, based on each piece of feedback information, that the data storage cluster storing the data to be migrated is the source data storage cluster, and to determine that the storage node storing the data to be migrated in the source data storage cluster is the target storage node; a routing sending module, which is used for the routing node to send the node identifier of the target storage node to the scheduling node; and a scheduling receiving module, which is used for the scheduling node to receive the node identifier of the target storage node, and migrate the data to be migrated from the source data storage cluster to the target data storage cluster based on the node identifier.

[0242] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the data migration method described in the embodiment of the present application.

[0243] The embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor will be caused to execute the data migration method provided by the embodiment of the present application, for example, Figure 5 The data migration method shown.

[0244] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various electronic devices including one or any combination of the above memories.

[0245] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0246] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0247] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0248] In summary, the embodiments of the present application have the following beneficial effects:

[0249] (1) In response to the data query request for the data to be migrated sent by the scheduling node, the routing node sends the data query request to the query node in each data storage cluster, and receives the feedback information returned by each query node for the data query request. Based on each feedback information, the source data storage cluster storing the data to be migrated and the target storage node storing the data to be migrated in the source data storage cluster are determined from multiple data storage clusters, and the node identifier of the target storage node is sent to the scheduling node, so that the scheduling node migrates the data to be migrated from the source data storage cluster to the target data storage cluster. In this way, the source data storage cluster storing the data to be migrated and the target storage node storing the data to be migrated in the source data storage cluster are determined by the routing node, and the node identifier of the target storage node is sent to the scheduling node by the routing node, so that the data to be migrated is migrated from the source data storage cluster to the target data storage cluster, so that the data service can operate normally during the migration process of the data to be migrated, and the data migration is realized without the data service being aware of it, so that the impact of data migration on the data service can be effectively reduced.

[0250] (2) When the feedback information carries the storage address of the data to be migrated, the storage address is compared with the address range of each storage node in the source data storage cluster to determine the target storage node from the storage nodes in the source data storage cluster; when the feedback information does not carry the storage address of the data to be migrated, a node query request is sent to the source data storage cluster, and the node identifier of the target storage node storing the data to be migrated returned by the source data storage cluster in response to the node query request. In this way, by determining whether the feedback information carries the storage address of the data to be migrated, different methods are used to determine the target storage node in the source data storage cluster, thereby achieving effective determination of the target storage node.

[0251] (3) By combining the remaining storage space of the storage node and the amount of data to be migrated, the target storage node is determined from the storage nodes of the target data storage cluster, thereby achieving optimal adaptation between the remaining storage space of the determined target storage node and the amount of data to be migrated, effectively improving the efficient use of the storage space of the storage nodes in the target data storage cluster, and effectively improving the utilization rate of the storage space.

[0252] (4) After data migration, applications that access data are unaware of it, which can greatly reduce the cost of application transformation and the risks that may arise during the migration process. By configuring the routing policy on the Router, the Router can route application requests to the HDFS cluster where data is stored. There is no need to modify the URI metadata in the Hive table after data migration. With the routing function implemented by the Router, applications no longer need to access Hive to obtain the URI, so this information does not need to be modified. New data writing is supported during the data migration process to avoid writing data during migration, which may cause inconsistencies between the migrated data and the original data. During data migration, the business side can still read files that have not been migrated, have been migrated, and have been newly written to ensure business continuity.

[0253] (5) Before data migration, after determining the source data storage cluster storing the data to be migrated and the target storage node storing the data to be migrated in the source data storage cluster, when the scheduling node has a need to read the data to be migrated, the routing node responds to the data read request for the data to be migrated sent by the scheduling node. At this time, the routing node already knows the source data storage cluster storing the data to be migrated. Then, the routing node can forward the data read request to the query node in the source data storage cluster. After receiving the data read request, the query node in the source data storage cluster reads the data to be migrated, thereby realizing effective reading of the data to be migrated before data migration.

[0254] (6) After receiving the data to be migrated read by the query node in response to the data read request, the data to be migrated stored in the source data storage cluster is deleted, and the routing node then migrates the data to be migrated to the target data storage cluster. This enables the storage space of the source data storage cluster to be quickly released during the data migration process, effectively reducing the data storage pressure of the source data storage cluster.

[0255] (7) After receiving the data to be migrated read by the query node in response to the data read request, the data to be migrated stored in the source data storage cluster is deleted, and the routing node then migrates the data to be migrated to the target data storage cluster. In this way, in the process of deleting the data to be migrated stored in the source data storage cluster, the data to be migrated is always retained in the routing node. When the routing node receives a data read request for the data to be migrated, it is not necessary to obtain the data to be migrated from the source data storage cluster and the target data storage cluster, but can directly read the data to be migrated from the routing node locally, thereby effectively improving the efficiency of obtaining the data to be migrated.

[0256] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A data migration method, characterized in that: A routing node applied to a distributed storage system, the distributed storage system comprising: a scheduling node, the routing node and a data storage cluster, the data storage cluster comprising a source data storage cluster and a target data storage cluster, the method comprising: In response to the data query request sent by the scheduling node, the data query request is sent to a query node in the data storage cluster, wherein the data query request is used to query whether the data storage cluster stores data to be migrated, and the data storage cluster includes a query node and a storage node; Receiving feedback information returned by the query node in response to the data query request, wherein the feedback information is used to indicate whether the data to be migrated is stored in the data storage cluster; Based on the feedback information, determining a data storage cluster storing the data to be migrated as the source data storage cluster, and determining a storage node in the source data storage cluster storing the data to be migrated as a target storage node; The node identifier of the target storage node is sent to the scheduling node, so that the scheduling node migrates the data to be migrated from the source data storage cluster to the target data storage cluster based on the node identifier.

2. The method according to claim 1, characterized in that The determining that a storage node storing the to-be-migrated data in the source data storage cluster is a target storage node includes: When the feedback information carries the storage address of the data to be migrated, the storage address is compared with the address range of the storage nodes in the source data storage cluster to obtain a first comparison result; Based on the first comparison result, determining the target storage node from the storage nodes in the source data storage cluster; When the feedback information does not carry the storage address of the data to be migrated, sending a node query request to the source data storage cluster, the node query request being used to query the target storage node among the storage nodes in the source data storage cluster; Receive the node identifier of the target storage node storing the data to be migrated, which is returned by the source data storage cluster in response to the node query request.

3. The method according to claim 2, characterized in that The determining the target storage node from the storage nodes in the source data storage cluster based on the first comparison result includes: When the first comparison result indicates that the storage address is within the address range of the corresponding storage node, the corresponding storage node is determined as a target storage node storing the data to be migrated.

4. The method according to claim 1, characterized in that After sending the node identifier of the target storage node to the scheduling node, the method further includes: In response to a data migration request for the data to be migrated sent by the scheduling node, forwarding the data migration request to a query node in the source data storage cluster, wherein the data migration request is used to trigger the query node to read the data to be migrated from the target storage node; The data to be migrated read by the query node in response to the data migration request is received, and the received data to be migrated is sent to the target data storage cluster.

5. The method according to claim 4, characterized in that Before sending the received data to be migrated to the target data storage cluster, the method further includes: Sending a storage space query request to the query node in the target data storage cluster, wherein the storage space query request is used to trigger the query node to query the remaining storage space of the storage nodes in the target data storage cluster; Receiving the remaining storage space of the storage node returned by the query node in response to the storage space query request, and determining a reference storage node from the storage nodes of the target data storage cluster in combination with the remaining storage space and the amount of data to be migrated; Sending a target node identifier of the reference storage node to the query node in the target data storage cluster, wherein the target node identifier is used to trigger the query node to send the data to be migrated to the reference storage node after receiving the data to be migrated; The step of sending the received data to be migrated to the target data storage cluster includes: The received data to be migrated is sent to the query node in the target data storage cluster, so that the query node creates an index entry for the data to be migrated and sends the data to be migrated to the reference storage node.

6. The method according to claim 5, characterized in that The determining a reference storage node from the storage nodes of the target data storage cluster in combination with the remaining storage space and the amount of the data to be migrated includes: Comparing the remaining storage space with the data volume to obtain a second comparison result; when the second comparison result indicates that the remaining storage space can accommodate the data volume to be migrated, determining the storage node as a candidate storage node; When the number of the candidate storage node is one, determining the candidate storage node as the reference storage node; When there are multiple candidate storage nodes, the candidate storage node with the smallest remaining storage space is determined as the reference storage node.

7. The method according to claim 4, characterized in that After receiving the data to be migrated read by the query node in response to the data read request, the method further includes: Sending a data deletion request to a query node in the source data storage cluster; The data deletion request is used to trigger the query node to delete the index entry of the data to be migrated stored in the query node, and send a data deletion request to the target storage node in the source data storage cluster, so that the target storage node deletes the data to be migrated stored in the target storage node; Receiving indication information returned by the query node in the source data storage cluster in response to the data deletion request, wherein the indication information is used to trigger the routing node to send the received data to be migrated to the target data storage cluster; The step of sending the received data to be migrated to the target data storage cluster includes: In response to the received indication information, the data to be migrated is sent to the target data storage cluster.

8. The method according to claim 7, characterized in that After sending the to-be-migrated data to the target data storage cluster in response to the received indication information, the method further includes: In response to a data read request for the data to be migrated sent by the scheduling node, forwarding the data read request to a query node in the target data storage cluster, wherein the data read request is used to trigger the query node to read the data to be migrated from the target data storage cluster; receiving the data to be migrated read by the query node in response to the data reading request; The data to be migrated is sent to the scheduling node.

9. The method according to claim 4, characterized in that The step of sending the received data to be migrated to the target data storage cluster includes: In response to receiving the data to be migrated, sending the data to be migrated to the target data storage cluster; After sending the received data to be migrated to the target data storage cluster, the method further includes: Sending a data deletion request to a query node in the source data storage cluster; The data deletion request is used to trigger the query node to delete the index entry of the data to be migrated stored in the query node, and send a data deletion request to the target storage node in the source data storage cluster, so that the target storage node deletes the data to be migrated stored in the target storage node; Receive indication information returned by a query node in the source data storage cluster in response to the data deletion request, wherein the indication information is used to indicate that the data to be migrated no longer exists in the source data storage cluster.

10. The method according to claim 9, characterized in that Before sending the data deletion request to the query node in the source data storage cluster, the method further includes: In response to a data read request for the data to be migrated sent by the scheduling node, forwarding the data read request to the source data storage cluster or the target data storage cluster; Receiving the data to be migrated read by the source data storage cluster or the target data storage cluster; The data to be migrated is sent to the scheduling node.

11. The method according to claim 10, characterized in that Before receiving the indication information returned by the query node in the source data storage cluster in response to the data deletion request, the method further includes: In response to a data read request for the data to be migrated sent by the scheduling node, forwarding the data read request to the source data storage cluster and the target data storage cluster; Receiving the data to be migrated returned in response to the data read request, when the data to be migrated returned by the source data storage cluster and the data to be migrated returned by the target data storage cluster are received, comparing the data to be migrated returned by the source data storage cluster and the data to be migrated returned by the target data storage cluster to obtain a comparison result; When the comparison result indicates that the data to be migrated returned by the source data storage cluster and the target data storage cluster are the same, performing deduplication processing on the returned data to be migrated to obtain target data to be migrated; The target data to be migrated is sent to the scheduling node.

12. The method according to claim 1, characterized in that The method further comprises: Sending a storage space query request to a query node in the data storage cluster, wherein the storage space query request is used to trigger the query node to query the overall remaining storage space of the storage nodes in the target data storage cluster; Receiving the total remaining storage space returned by each of the data storage clusters in response to the storage space query request; The target data storage cluster is determined from the data storage clusters based on the overall remaining storage space.

13. The method according to claim 12, characterized in that The determining the target data storage cluster from the data storage clusters based on the overall remaining storage space includes: Compare the overall remaining storage space with the amount of data to be migrated to obtain a third comparison result; when the third comparison result indicates that the overall remaining storage space can accommodate the amount of data to be migrated, determine the data storage cluster as a candidate data storage cluster; When the number of the candidate data storage cluster is one, determining the candidate data storage cluster as the target data storage cluster; When there are multiple candidate data storage clusters, the candidate data storage cluster with the smallest overall remaining storage space is determined as the target data storage cluster.

14. A data migration method, characterized in that: A scheduling node applied to a distributed storage system, the distributed storage system comprising: the scheduling node, a routing node and a data storage cluster, the data storage cluster comprising a source data storage cluster and a target data storage cluster, the method comprising: Sending a data query request to the routing node, so that the routing node sends the data query request to the query node in the data storage cluster in response to the data query request, and receives feedback information returned by the query node for the data query request, and based on the feedback information, determines that the data storage cluster storing the data to be migrated is the source data storage cluster, and determines that the storage node storing the data to be migrated in the source data storage cluster is the target storage node; The data query request is used to query whether the data to be migrated is stored in the data storage cluster, and the data storage cluster includes the query node and the storage node; In response to the node identifier of the target storage node sent by the routing node, the data to be migrated is migrated from the source data storage cluster to the target data storage cluster.

15. The method according to claim 14, characterized in that The step of migrating the data to be migrated from the source data storage cluster to the target data storage cluster in response to the node identifier of the target storage node sent by the routing node includes: In response to the node identifier of the target storage node sent by the routing node, a data migration request for the data to be migrated is sent to the routing node, so that the routing node forwards the data migration request to the query node in the source data storage cluster, receives the data to be migrated read by the query node in response to the data migration request, and sends the received data to be migrated to the target data storage cluster; The data migration request is used to trigger the query node to read the data to be migrated from the target storage node.

16. The method according to claim 14, characterized in that The method further comprises: Sending a data read request for the data to be migrated to the routing node, so that the routing node forwards the data read request to the query node in the target data storage cluster, and receives the data to be migrated sent by the query node, wherein the data read request is used to trigger the query node to read the data to be migrated; The data to be migrated sent by the routing node is received.

17. A data migration method, characterized in that: Applied to a distributed storage system, the distributed storage system includes: a scheduling node, a routing node and a data storage cluster, the data storage cluster includes a source data storage cluster and a target data storage cluster, the method includes: The scheduling node sends a data query request to the routing node, where the data query request is used to query whether the data storage cluster stores data to be migrated; The routing node sends the data query request to a query node in the data storage cluster in response to the data query request, and receives feedback information returned by the query node in response to the data query request, wherein the data storage cluster includes the query node and a storage node; The routing node determines, based on each piece of feedback information, a data storage cluster storing the data to be migrated as the source data storage cluster, and determines a storage node in the source data storage cluster storing the data to be migrated as a target storage node; The routing node sends the node identifier of the target storage node to the scheduling node; The scheduling node receives the node identifier of the target storage node, and based on the node identifier, migrates the data to be migrated from the source data storage cluster to the target data storage cluster.

18. A data migration device, characterized in that: A routing node applied to a distributed storage system, the distributed storage system comprising: a scheduling node, the routing node and a data storage cluster, the data storage cluster comprising a source data storage cluster and a target data storage cluster, the device comprising: A request sending module, configured to respond to a data query request sent by the scheduling node and send the data query request to a query node in the data storage cluster, wherein the data query request is used to query whether the data storage cluster stores data to be migrated, the data storage cluster includes a query node and a storage node, and the multiple data storage clusters include a target data storage cluster; A receiving module, configured to receive feedback information returned by each query node in response to the data query request, wherein the feedback information is used to indicate whether the data to be migrated is stored in the corresponding data storage cluster; A determination module, configured to determine, based on the feedback information, a data storage cluster storing the data to be migrated as the source data storage cluster, and to determine a storage node in the source data storage cluster storing the data to be migrated as a target storage node; The identification sending module is used to send the node identification of the target storage node to the scheduling node, so that the scheduling node migrates the data to be migrated from the source data storage cluster to the target data storage cluster based on the node identification.

19. A data migration device, characterized in that: A scheduling node applied to a distributed storage system, the distributed storage system comprising: the scheduling node, a routing node and a data storage cluster, the data storage cluster comprising a source data storage cluster and a target data storage cluster, the device comprising: A request sending module, used to send a data query request to the routing node, so that the routing node sends the data query request to the query node in the data storage cluster in response to the data query request, and receives feedback information returned by the query node for the data query request, and based on the feedback information, determines that the data storage cluster storing the data to be migrated is the source data storage cluster, and determines that the storage node storing the data to be migrated in the source data storage cluster is the target storage node; wherein the data query request is used to query whether the data to be migrated is stored in the data storage cluster, and the data storage cluster includes the query node and the storage node; The response module is used to migrate the data to be migrated from the source data storage cluster to the target data storage cluster in response to the node identifier of the target storage node sent by the routing node.

20. A data migration device, characterized in that: Applied to a distributed storage system, the distributed storage system includes: a scheduling node, a routing node and a data storage cluster, the data storage cluster includes a source data storage cluster and a target data storage cluster, the device includes: A scheduling sending module, used for the scheduling node to send a data query request to the routing node, wherein the data query request is used to query whether the data storage cluster stores data to be migrated; A routing response module, configured for the routing node to send the data query request to a query node in the data storage cluster in response to the data query request, and to receive feedback information returned by the query node for the data query request, wherein the data storage cluster includes the query node and the storage node; A routing determination module, used for the routing node to determine, based on each of the feedback information, that the data storage cluster storing the data to be migrated is the source data storage cluster, and to determine that the storage node storing the data to be migrated in the source data storage cluster is the target storage node; A routing sending module, wherein the routing node sends the node identifier of the target storage node to the scheduling node; The scheduling receiving module is used for the scheduling node to receive the node identification of the target storage node, and based on the node identification, migrate the data to be migrated from the source data storage cluster to the target data storage cluster.

21. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; A processor, used to implement the data migration method described in any one of claims 1 to 17 when executing the computer executable instructions or computer programs stored in the memory.

22. A computer-readable storage medium storing computer-executable instructions, characterized in that: When the computer executable instructions are executed by a processor, the data migration method described in any one of claims 1 to 17 is implemented.

Images